Why is my environment mesh collision so off?

That looks like auto-generated collision.

What you’ll probably want to do is make modular building meshes, like walls, floors, doors, etc. with accurate simple collision in Blender. Then import the modular meshes and build out houses, tunnels, and stuff in Editor.