I think you need to check your lumen settings and nanite/mesh settings too.
You are right that the scale of your actors is causing the problem, but that is because lumen works with Mesh Distant Fields and NOT Actors distant fields. Since your meshes are scaled 100x lumen is still using MDF of your 1x mesh. So first i would try making your walls/ceiling 100x larger and not scale them. If that doesn’t help, also check in your project settings the Distance Field Voxel Density value. changing this will recalculate all of your distance fields so it would be better to make a test map with only 1 test room.
Then check in the mesh editor the distance fields values of your meshes too.
You should definetely read this Lumen Global ilumination and this Mesh Distance Fields