Strange skylight effect. HELP

I am here giving a link to Dropbox with 3 zipped files, you can choose any of them to download.

testhouseFULLmaterials - All the materials in the project
testhouse - Only walls, floor and stairs
testhouseNOmaterials - No materials at all

If on any of them if you use the Lighting Only Viewmode they all look the same with the same noise.

So just to make sure the problem wasn’t the sphere I created I added from the Engine an HDRi Backdrop and changed the cubemap to the one I had.
I applied the same settings and I got the same results, the same noise.

I kept tweeking the lightmass and intensities to try and minimize the noise and its almost gone. My Directional Light was always Off, I was working only with the skylight and HDRi.
Once I build using the Directional Light not much changed regarding the noise.

Just for the reference I am using UE 4.25.3, GPU Lightmass, the RayTracing is Off and DirectX12