It is invalidating the shadow cache pages first cvar is a fix for that, if you are doing anything with camera FOV see the last cvar
r.Shadow.Virtual.Cache.MaxMaterialPositionInvalidationRange 10
aaand after 2 sleepless nights
r.Shadow.Virtual.NonNanite.IncludeInCoarsePages 0 (if you have alot of non nanite stuff casting shadows)