Showing a sub-level changes fog/lighting of parent world!

I am making an open world game, it has a large terrain with small ‘towns’ contained in sub-levels, these sub-levels appear and disappear as you get close.
The issue is that when a sub-level appears it seems to have a drastic effect on the world lighting, sometimes it can make everything seem brighter, like the ‘skylight’ has been increased, and the fog can become significantly denser/brighter too.

I thought it might have been that the sublevels are containing additional skylights or directional lights but the problem still happens when there’s no lights or fog actors in the sub level at all!

I think it must be something to do with the sub-level’s lighting build. The lighting for each sub level is built individually.

What is happening? How can I make sub-levels not affect the whole-world lighting/fog like this?