Having issues with Physics constraints of attached components

Hi everyone. I am trying to use a physics contraint to unsheathe a katana from its scabbard in a VR game and does not maintain it’s relative location when I grab the katana (attach to hand mesh). Here is an image of the issue:

When I drop the blade, the scabbard constrain keeps working relative to the hand mesh and not the blade.

Note: I used a cylinder with modified scales as scabbard (I have not done the model yet).

The hand mesh has a physics constraint with the MotionController as it follows:

I am using the VR Template as a base. Therefore I have used the PickUpInterface to grab objects. In the specific case of the Katana, the PickUp Event does the following:

I had to detach everything before doing attachments because the Hand Constraint gets crazy otherwise.

Finally, this is my Katana config:

I have tried to do the grab system through only physics constraints, but I have better results attaching the Katana to the Hand Mesh and then applying the constrain to the whole group. With only physics constraints I got a “laggy” movement of the blade.

Thanks in advance for any guidence!