How much work is it to replace the physics engine?

That is actually true.
And physical objects could probably be read from the level and feed to the “personal” physics engine, making object sizes and shapes at least for the static level be provided from the editor if desired.
Why didn’t I think of this.
I am actually doing something similar controlling actors by fixed positions trough logic, rather then physics, and just use parts of the physics implementation like rays to know about the static world.