How do I get rid of this ghosting caused by Lumen GI?

There are no shadows, as in no direct shadows? Lumen is only (for now) an indirect lighting solver, that is a very strange phenomenon.

What mode is lumen set to? Are you operating on Shader Model 5 or 6?