Actors are flying away when I get close and not allowing me to move when I pick them up in VR

I seem to be having a collision issue with the VR Pawn and the actors in my scene. When I pick up an actor and try to move, I seem to be getting pushed back. There is also a point where the actors sometimes fly away from when I get close to them. The only time I am able to move with actors in hand is when I move the actors away from my body towards the sides. What am I missing? I attached a video for anyone to see the issue.