Weird Skylight artifacts with Lumen

Im having big time issue with setting up a Skylight lit scene - it basically creates this insane artifact that is impossible to get rid of when using Lumen.

Any tips/tricks to resolve this?