I found out that it is was caused by having Extended Structures enabled. Without it, I can damage everything. Although having it disabled causes them to “unhinge” and jitter into each other. Anything I could do here?
I also found out what the cause was for the player being able to destroy the mesh by running into it. All my objects still had Simulation Generates Hit Events enabled, which is not needed anymore, since I now use line trace.
It’s looking really good already, thanks a lot! 
(You can see the wall unhinge awkwardly from the one next to it in the top right.)