Why are objects in the shadows being highlighted?

I’m trying to get some nice lighting in my scene but all sides that are at a certain angle are being highlighted either by the skylight or global illumination.

Here’s skylight only:


And here’s built lighting with a skylight:


The ground is green and it’s far below. But no matter how occluded the side is it still has a green tint and it’s bright:


Does anyone know how to fix this issue?

Thank you.