Set physics constraint target orientation to look at world object

I’m making a VR game, although this problem is not VR specific.

I have an actor attached to the motion controller using a physics constraint. So in the constraint, component 1 is a physics object that follows the transform of the motion controller (player’s hand), and component 2 is the root component of the object being grabbed.

I want to set (every frame) the angular target orientation of that physics constraint to point at the second motion controller.

And basically I’m lost, I’ve been trying the whole day to do that using findLookAtRotation and findRelativeLookAtRotation, nothing works, I can’t even get it to look consistently at any point in the world.

For a simpler test I attached a static mesh to the object I’m grabbing and I managed to have it point to the motion controller consistently, regardless of the rotation of the actor I’m holding:

However, when I do the same by setting the target orientation to the constraint, the object I’m grabbing is constantly spinning.

I assume this has to do with the actor’s rotation changing every frame but I can’t figure it out.

Any help would be appreciated.