grabbing objects using the "Grab Component at Location and Rotation" system?

Currently i’m working on a VR game that allows players to grab objects for interaction.

Initially i thought i could do with the “attach to component” system with physics disabled, however this provides infinite strength.
The result of this is that the player is capable of manipulating items through other items. This looks glitchy and broken.

Now i’m working on using the GrabComponent together with the physics handle component. However i cant seem to get it to follow the rotation of my vive controllers properly.
I’m using the “set target rotation” to set the rotation on each tick.

This node works great for keeping the object steady at it’s initial rotation when it was first grabbed, however rotating the controller does not affect the object obviously.

Currently i’ve tried various variants of combining/subtracting the “current” rotation of the object/controller and the position during the pickup tick (see the stored rotation variables) Though for some reason i cant seem to get the rotation to follow correctly in all axes.

Is there a proper way to combine these (or other) variables to get a proper offset/rotation behaviour?