Thanks @Klian and @LoPing for the great insights.
I follow your ideas and found a solution:
- I changed the plan to a cube, and realign them to intersect in the corners.
- I changed the collider preset to FortDynamicMeshPhysics.
With both changes, now it works fine.