Enable and Disable Grab Component for mining experience

I am working on some mechanics for VR mini experience that will work on the quest with unreal 4.27.

You pick up a pickaxe, you stike it on the wall a few times, and you pick up the materials and put them into a kart.

In my current test, I have about dozen cubes making up a section of the wall. After they are struck 3 or 4 times, physics will activate, and they will fall to the ground.

The problem I am having is picking up the items. When I add a grab component, everything else seems to break. Physics activates early and causes other glitches. I need to turn it off up until the physics has been activated. But I cannot work out how.

This setup is “fine” I can technically say it works, and it will suffice for what I am trying to do. I will be interested if you have any thoughts on how to improve. I explored the fracture object, though that never seemed to activate.