Yeah if you want the hand to not have that behavior on specific objects then you will need to change its logic to have alternatives, also likely increase the strength of the constraint on the body as that appears far too weak to actually drag the body (disconnecting your hand). Also the bodies collision shapes appear to be self colliding causing instability as well.
You could use a hard constraint to the hand as well, but you won’t get soft interactions with that.
If it was the other way around and the mesh attached to the hand but the hand remained jointed it would be no different with the constraint strengths and broken collisions you have there…