The grasping hands already don’t attach until lerping to hand is done, sounds like you are gripping at a distance without lerp to hand so it just attaches right away. Letting the object fly to you just with the constraint is likely not the best idea, i would enable the GlobalLerpToHand settings in the project settings.
Otherwise if you don’t want to turn that on, you can modify the grasping hand blueprint graph to attach when the object is within a set distance to the grips target transform instead of right away when the OnGrippedObject even is thrown.