Also still curious how lighting and shadows becomes more of a problem when actors not visible or casting their shadows are populating the world outside. I am perplexed.
I’m also not completely clear on that, but I know that every single object in the game has at least some impact regardless of if you can see it or not because the engine has to figure out if it can see the object or not every single frame. Maybe it does the same for every shadow?
It occurs to me that there is also an option inside each light to control the distance at which it is active. You could try adjusting that to see if that helps fix some of your issue as well.
Anyway, you know what the issue is, and you are working on a solution. Good luck with your game!