Vive and Character with physics

Hello there. I’m developing a game for the Vive in which the player uses a jet-pack. In the set-up I’ve got right now (I tried some others) there is the JetpackPawn (no collision), which has 2 motion controllers and every tick figures out how much force to apply to a second actor, JetpackPhysics, which just contains an sphere collision with ‘Simulate Physics’ on. Then, I move JetpackPawn to JetpackPhysics. This works pretty well, but I’m running into a problem:
When going at high speeds, the motion controllers move like crazy. This also happens if I do all physics inside the JetpackPawn.

Any ideas on how to solve the issue? Is there a better way to approach this problem?