Use motion controller to move view

I’m trying to use the motion controller to slide around the view, as a stepping stone to working out my own climbing implementation in VR. The problem is that I’m always pushed around just a little bit every time I use the grip button. The grip works perfectly after this push, but the second I click the button, I shift a bit. The amount seems based on the relative position of my hand when I try to climb, so I know it has something to do with relative location, but for the life of me I can’t figure out what I’m doing wrong. I’ve been swapping pins for hours with no effect. What am I overlooking??