That’s one of the limitations of Lumen in some light setups like yours.
It uses screen space with raytracing as Screen-Tracing to generate the lights and shadow bounces.
As you move your character/camera sometimes you will see pop ins for both light and shadow.
If you need very fine tuned approach maybe bake those lights? Or go full ray tracing