Found one more thing, maybe the noise is there only with Hardware RayTracing. Maybe is the contrast that is causing this, dunno. Sad that I lose a lot of the image quality.
I am just confused, if I turn down AA quality i can reduce the noise with hardware raytracing and maxed out the other settings, seems a good tradeoff.