Why is my project performance worse compared to 5.3.2 when the opposite was supposed to happen?

There are several who have reported the same. Nothing has changed in the project and we have 2 identical builds one with 5.4.1 and the other 5.3.2 - the 5.4.1 version has on average 20% worse performance, when the expectation was to have a performance gain with the new version. We use Nanite and Lumen.

Are there some new default setting that are heavier or enable something we are not aware of? Any similar experiences?