I am having very similar problems, and they are very persistent.
I have tried updating my graphics drivers, swapping to dx11, reinstalling UE 5.1, unplugging and plugging everything back in, and reinstalling windows before trying those steps (and more) again.
The behavior does not change. I get very inconsistent results when attempting stereo rendering in the VR template, and in a blank project. These results are mostly crashes with no report generated. Occasionally, another report will generate. Other times, certain shaders fail to load. The strangest result was a mostly successful render to my HMD, which was only unworkable because the world was upside down.
I hope I’m missing something basic, but it seems like my UE is very haunted and I must wait for a fix. I have not tried on another computer, because I don’t have access to another one with a DP port to plug in my HMD.
I will be borrowing an Oculus Quest and trying on another computer soon.