As of now my displacement map is correct when I added the Smooth operator to my mesh (like the screenshot above).