Skylights can’t be dynamically shadowed by meshes, so as far as it’s concerned your walls don’t exist. I’m guessing that’s the source of your strong ambient lighting.
You’ve got a few approaches for skylight shadowing.
Shadowing from lightmaps (this may or may not work for your scene. Why do you need to disable baked lighting?)
Shadowing from ambient occlusion (you could bake this to a texture map or vertex colors)
Shadowing from distance fields (this needs to be enabled in the project settings, and has a performance cost. There’s a link at the bottom.)
Alternatively, you could alter the content of your skylight to match your desired scene lighting, or replace it with a post-process lighting feature.
This all assumes that the skylight is the culprit. What does the scene look like with it disabled?