Since lumen is an accumulation based approach, it is going to struggle with dis-occlusion. Depending on lighting conditions, I find it can take up to 4-8 frames to resolve a clear and stable image.
Decreasing temporal accumulation will only make the rest of the frame as bad as the dis-occluded part.
The only way to really fix it is to improve the spatial quality of Lumen so that the image is clearer on frame one. It can also be improved by running a faster frame rate, so that the 4-8 frames have less persistence on screen.
Tech such as Nvidia Ray Reconstruction will probably help this case too.
Strafing against high parallax scenes is also a worst case scenario for this kind of issue.