Do you want to move everything except the player? In that case it’s easier to move the player in the opposite direction instead.
In general this is done by storing the position of the hand when you start grabbing. Then every frame you take the original position, subtract the current position of the hand from it, and move the player pawn by the resulting vector.