Virtual Shadow Maps causing huge leaks

Blocking out a simple scene in 5.1 and I’m getting huge light leaks and flickers. No gaps in the mesh, even when I put the player start into a room with no openings I’m still getting huge light flickers from the outside Directional light. Turning off Virtual Shadow maps in the project settings instantly fixes the problem, but I’d still like the soft shadow advantage of the VSM. I’ve increased the light map resolution to 128 or 512, makes no difference. I’d love to know how to fix this without just shutting VLM off.
Thanks