I just tested it again, I didn’t know at first that when grabbing the object it grabs it by the mesh. I assumed it was the grabbing the collision volume of the object (like the blue UE4 vr box).
When I was grabbing the object quickly, the orb was passing through the mesh giving me the impression it was not grabbing correctly.
I can work around by adding handles to the objects, also helps indicate where to grab.
However is it possible to grab the object by its collider or volume? My first impression is that “GrippableBox Component” may be the answer?