How to have skeletal mesh interact with Chaos / Geometry collections

I’m working in unreal 5.2 on a VR project where I would like the player’s hands to be able to destroy buildings. With a fresh Geo collection everything works fine, but because the destructibles are buildings I would like the base to stay in place. However, once I use either an anchor field or turn certain pieces kinematic/sleeping, all collision with the skeletal mesh stops and no destruction happens. Any tips? There’s not a lot of good resources on Chaos sadly.