It looks like you’re not accounting for the HMD orientation and offset correctly within your characters AnimBlueprint. I could be wrong, but that’s my intuition. The shoulder or spine transform is not rotating with the head, so there’s probably a transformation break in the AnimGraph.
Does the out of the box version work perfectly? If so, does creating a character which inherits from the OOTB version break? If not, does adding a material billboard break it?