Do you have to move your axis in translation inside the hole?
I would create a physic constraint with all translation blocked and only a rotation free.
If the two collision shapes collide you might also want to disable the collision on the constraint.