I'm having a lighting issue possibly related to LOD but no changes I make seem to fix it?

I have a directional light that is hooked up to my sky sphere . The issue I’m having is whenever the sun goes down some of the ground mesh is still illuminated. It appears to be related to the distance away from the camera, I’ve played around with some LOD settings but I haven’t been able to change anything.
Some other important details is that my map was not made with the landscape tool, it is generated in blueprints and the meshes are HISMs.

These first three pics are what the ground looks like from three different distances. You can still see the outer walls are all dark. (also ignore the bright sky sphere in the last image)



Below are two more pictures looking out across the map, one from the viewport and the other is what you see when you play the game at night.


It’s definitely related to the distance from the camera but I can’t figure out why it’s happening or how to fix.