Lumen - Global illumination inconsistent behaviour

I am getting really inconsistent results when using a camera to fly around some items and buildings. You can see in my demo, that the sides of the cubes are getting darker and brighter when moving the camera.

In this this test scene there is one direct light with default settings - except that “Cast shadow” is turned off, so we don’t mistake this for a shadowing issue. Meshes are just standard in cubes and plane. The entire project has default settings. using Unreal 5.3.2

It doesn’t help cranking all the Lumen-settings to the max, still getting the same issue. Also rendering with many samples. It’s not an antialias TSR issue, turning AA to none makes no difference.

I feel that I am missing something here, there is some setting(s) or way to do things that I am not using. In my real project, an animation, this issue causes very flickery weird artifacts all through the animation, even if I turn all rendering settings on max with a carefully built scene.

It looks like the light GI is kind of lagging behind when moving around.

Does anyone know how to fix/improve this problem, either with some settings or console commands?