How to rotate grabbed object with player camera and original offset?

Hi, Im trying to make a grab item system where you grab the object and it rotates with the player camera as your turn but keeps the original offset of its grab. However despite using world transforms it only behaves correctly when grabbed from one side. If grabbed from the other faces there is a wierd initial rotation.

My calculation is essentially:
Offset = mesh world rotation - camera world rotate (Using delta rotate node)
Set mesh world rotation per tick to Camera world rotation + Offset using Combine rotator

Heres youtube vid:

Here is highlighted sections of problem blueprint code: Rotate object issue - Album on Imgur
Here is blueprint link for object code: Unreal Object Grab posted by anonymous | blueprintUE | PasteBin For Unreal Engine