Issues with VR Hand collision

Hey, hobbyist dev here trying to figure things out as I go! So forgive me if I don’t understand the correct terms and what not!
I’m trying to create a telekinesis system that takes the mass of the component into account, which I’m happy with overall. Some tweaks here and there maybe, but I think that’s good.
The issue I’m having is that I’m able to hit the actors as if they have no weight at all using my hands.

Where would I need to start to change this behavior? I’d rather not disable the collision on the hands unless there is no other option.

Here’s a video to showcase what I mean