moved this since it was better as a comment:
Been fighting this today. There is a BP node “Enable Player Controller follow HMD” that is also causing the camera to follow the headset. Using this node to disable controller following will stop the position from changing w.r.t. to the VR system. It also seems to disable any control of the character at all using VR however. I’ve also been playing with the “Enable Player Camera Manager Follow HMD” node… with no positive results