Developing unreliable with Oculus Rift?

I’m currently working on a project regarding virtual juggling using Oculus Rift and tracking markers on the arms to simulate throws.

I have an USphereComponent that works as the ball and is moved to the hand when holding it by turning physics off(once) and setting the location of the ball to the hand every tick.
This means the hand has no collision at all and doesn’t affect the ball in other ways than how I choose to manipulate it.

For throwing the ball, the physics is turned on and the ball is given an impulse in a direction and speed derived from the hand.
I multiply the throwing-vector with FVector(0, 0, 1) so that the ball should only be thrown straight up just as a test.

This worked fine WHEN NOT USING VR, but when I pressed alt + enter in standalone so I go into VR mode, it throws in all directions!! Completely reproducible just by entering or exiting VR. In normal mode ball is thrown straight up.
So I tried multiplying throw by (0,0,0) (or just commenting out the impulse), so when you throw the ball, the only thing manipulating the ball is turning physics on.

In normal mode the ball is released with no speed and starts getting pulled by gravity, going through the hand and STRAIGHT down regardless of hand motion.
In VR mode the ball is thrown slightly in the direction of the hand when relasing before starting to fall down, WITH NO IMPULSE OR FORCE GIVEN TO THE BALL IN CODE.

I can literally just press alt + enter to change the program, how is this possible and why does it happen?
I also tried different modes, like VR preview gives same result as VR from standalone.

TLDR; A ball is not being thrown in the code, thus not being thrown in the program. However, when VR-mode is on the ball is being thrown. Why?