Any time I have more than one light in the scene, the left eye of my HMD has shadows flickering on and off - I’ve attached a video showing it. Happened in versions both 5.3 and 5.4.
I had the same issue! To solve it I changed the Virtual Shadow Map to Shadow Map setting in Project Settings / Engine / Rendering / Shadows
I hope it helps!
This isn’t a solution though because nanite relies on virtual shadow maps…
update: tried it and now the the shadows move with my headset in my left eye, and in the right eye everything is fully shrouded in shadow.
Does the problem go away if you use r.shadow.virtual.onepassprojection 0 in the console? (May degrade performance if you have many lights)
Was having the same problem as OP and this worked for me. Thanks @wouterdek-dev!
Have you found a solution for this problem? i think it started to occur since 5.3, I need to keep virtual shadows on disabling nanite is a nono 
have you tried entering r.shadow.virtual.onepassprojection 0 ?