As I can see it you only have one problem - the hand actor is not interacting with your object.
The casting thing is understandable since we are poking it with other objects which are not supposed to work anyway.
Unfortunately I don’t have that much experience with VR. However, I found a thread with a similar problem: How to use box collision with VR hands on UE5?
Hope it helps…