Lumen is temporal, it works by tracing(the expensive part) a fraction of pixels and rotates through the fraction over time every frame to save performance. If we trace all pixels for lighting every frame, the GPU would have a hard time remaining responsive.
It then uses TAA to smooth any noise from alternating tracing directions as well.
It’s “normal”, but the Lumen devs have no solution as of now.