Why displacement maps cause noise in Quixel textures?

I am trying to make a stone road using different material from Quixel collection, but applying displacement on Nanite landscape causes it to have a very strange look up close.
Screenshot 2024-02-04 114035

I tried on different materials and textures, it have this kind of things every time. I would like to be able to create a road using a texture instead of laying it out with mashes, but cant figure this out.

It looks better from a farther distance, but most of videos on youtube try not to zoom close to show actual effect mostly.

Screenshot 2024-02-04 114057

My material setup is very simple.

I would appreaseate help here very much, as i spended all day yesterday trying to figure this out )