The skylight will affect everything unless you’re using something more complete like Lumen or baked lighting. There’s also DFAO which will darken larger areas occluding the skylight, but it’s a more rough approximation.
So your options are:
Use Lumen
Use baked lighting
Turn off the skylight if in occluded areas (or change it to a different HDRI that better represents the scene)
Use DFAO and deal with some of the effect
Use something else for lighting