For anyone with similar issues, I found a solution.
This is the math for grabbing something and rotating the camera around it. You need to have a parent for real time rotation, and on release transfer the rotation to 1 child lower in the hierarchy, then reset rotation to the parent like "Real time billboard → “Rotation paste billboard” → Camera+Hand
(e.g. while grabbing, rotate parent billboard caller “real time billboard” then on release, paste rotation to 'real time billboard’s child called “Rotation pasting billboard”. Then reset Real time billboard’s rotation to 0.0.0
