Are you using a landscape?
In affirmative case, try to turn on “Shadow two sided”, it will block the light from below your landscape and could change the result a lot:
