lumen shadow problem (issue like ghost shadow)

When I set lumen and use my own model to build such an indoor environment, I get this strange changing ghost shadow, how can I solve it (I found the problem is not that hard when I use the standard cube mesh from Unreal)

Also looking for a solution to this!