Controller rotation is passed on to next level in VR and can't be reset.

When I load level 1 my character will face the correct direction and everything is fine. But when I load level 2 the current controller rotation is passed on and I’m looking the same direction as I was on level 1. This is not very good because I need the actor to read a text in front of them at level 2.
If I launch the exact same game but I don’t go to VR mode then the controller rotation is fine on level 2 and the player will have the text in front of them.

I have tried the “Reset Orientation and position” and setting yaw to zero at level 2 start. Still no go.
I have also assigned a keyboard key that sets the controller rotation to zero when pressed. This actually resets to the correct direction when pressed, but only when in non-vr mode. In VR nothing happens.
I have even tried creating a new separate controller for level 2 but it’s still the same problem.

Any idea what is going on?