Physics Control makes the mesh shift!

Physics Control makes a mesh shift while physics.

As you can see, the hands here are shifting from the pole as the character swings. The higher the speed or lower the FPS, the greater the shift!

Any idea how to fix this?