Here’s how I managed to do it:
1 - Go to ProjectSettings->Collision->Preset and create a new preset, use the same settings as the Destructible profile, with the exception of the Pawn = Ignore (I created it with the name “DestructibleNoPawn”)
2 - In the level viewport, select your chaos geometry collection, go to the Details panel, find the CollisionProfilePerLevel, Add two entries, the first one should be “Destructible” and the second one should be the new one you just created, like “DestructibleNoPawn”