Help with physics constrain issue

Setup
I am working In VR and Im picking up physics objects with physics constraints, My hands have a physics collision box around them.
I’m using a Timeline & lerp transform to make the objects fly in to my hands from distance.

Issue
Im setting the constraint once the objects have traveled to my hands, after the timeline, But as disable collision between objects in the constraint wont be enabled until after they have traveled to my hands, they hit the collision box around my hand and prevent the objects from sitting in them correctly before the constraint is applied giving me inconsistent Grip results.
So on grab I decided to disable the collision between my hand and the object and re-enabling it after the constraint has been set as I still want my other hand to be able to not pass through the held object.

Is there a more graceful solution to this , Like being able to set the constraint first then move the item to my hand and have that be the constraint position? and if so what Do I need to do?