VR hands with collisions

Sup folks

Im trying to implement a VR hands which will have collisions with physical objects and static meshes.
Unfortunately i didnt find any tutorial/course related to this topic and really struggle with that task.
My goal is to have some amount of predifined collision (since it looks like dealing with phys.asset would be more complicated).
So 1 collision for Idle hand position, 1 collision for Pointing, one collision for Fist.

Things which im currently try:

Adding phys. asset to VR hands works, but its complitely ignore AnimationBlueprint.

I was try a bit more “hacky” way:

My idea was to simply prepare 3 different static meshes with different poses and collision and simply switch between them.
So after the animation of hand, as example, from idle to pointing are ended, i switch idle collision mesh to pointing collision mesh.

It kind of works, but each time i switch it, physics start to behave wierd and hand became offsetted and never going back to initial position.