How to set Center of Mass so it aligns with the scene root's pivot

I am trying to physically control a hand actor via spring model connected to a Steam VR motion controller for a VR project.

Basically I assign a linear velocity to the hand actor with respect to the distance vector between the hand and the respective motion controller which I scale by some stiffness coefficient. This works fine and the hand translates towards the controller without any problems.

I am trying to do the same with angular velocity. I find the shortest rotation (quaternion) which rotates the hand actor towards the rotation of the controller so that they align to each other. I converted this to axis-angle notation and calculated the angular velocity that would rapidly orient the mesh to the controller (again scaled by some coefficient).

My problem is no matter how I set the center of mass offset off the hand, it still translates (not just rotates around its center) as I apply SetPhysicsAngularVelocity. It looks like as if it is doing orbital motion around the controller when the user slightly rotates the controller.

The hand actor is a blueprint actor which has a static hand mesh (which is not rendered and only used for physics) as its scene root and two more child components (one for hand’s metal bracers (rendered static mesh) and one rendered skinned mesh for hand animations.) The metal bracers component is set to auto-weld and is not simulating physics. So does the skinned mesh.

Is there a valid solution to this problem?

Thanks in advance.