Hello,
In order to simulate physics, the object will need proper collision. BSPs made into meshes do not have this by default. Did you create collision bounds for the mesh? You can go into the Static Mesh editor, then click the “Collision” dropdown at the top, then click 6DOP Simplified Collision to give it a box-shaped collision. After the collision is added, save the mesh and turn on physics. Please let me know if that helps.
Cheers!