GPU performance hit when moving camera - Slow virtual shadow maps?

Hey all. VSM has respectable performance across our project, but we see a sizable performance hit whenever the player pans the camera rapidly over the environment. We’re talking a difference of 10ms GPU thread to 22ms GPU thread. It rapidly adjusts and levels out, but it makes camera movement feel much less smooth and fluid.

I believe VSM are generated on screenspace and so are regenerating as the camera is moved, is there some settings I can change somewhere to account for this? I know lots of rendering settings have defaults set for close-perspective games (3P, FP), and ours is a top-down perspective.