Constrain grabbable VR object properly

I don’t know if this can help, it works only with controllers though. Best of luck!