VR - Camera and Controller motion adding impulse

So I’ve got a kind of sliding jetpack type game going on with the Oculus Rift.

I am using AddForce to move the player capsule. The problem comes with this not working every time unless ‘Simulate Physics’ on the capsule is checked.

The problem with ‘Simulate Physics’ is that moving the controllers or moving your head will add a force that will move the character around.

Is there any way to have AddForce work every time without ‘Simulate Physics’ or to make it so with ‘Simulate Physics’ moving the headset or controllers suddenly do not make the character jump around?