The problem is that if the person in the headset has moved physically in the room, the location they end up at is offset from the target location by seemingly the same distance…
The code applies the local space offset of the camera to the world space actor coordinates. If the player pawn is rotated, the coordinate system directions will not match anymore. The VR template has an example on how to apply the pawn rotation onto the coordinates.