Need help with attaching a basic mannequin to a physics handle

In my project a lot of the gameplay will be centered around a mechanic involving picking up people from a top down perspective, to either throw them away or place them at a job (black & white style) I have managed to set up the a hand which traces a line to a bone, or component. When I pick up the basic cube, no problems, but when I hit a bone on our basic man character, it is hitting the correct bone, and lazily dragging him by the bone, no problems with the hit. if I move quickly it swings him around at high speeds, and moving slowly it just lifts the leg or part that I’ve grabbed, without moving up to the hand to attach, as if the skeletal mesh was too heavy. I’ve tried a few interp settings, and a few dampening etc settings but he doesn’t seem to want to cleanly attach to the hand while he goes into physics simulation.

Any advice on some things I can change to get him to cleanly dangle from the handle without rubberbanding and acting very heavy?