Ref: “World Displacement works very much like World Position Offset, but it uses Tessellation vertices rather than the base vertices of the mesh”.
I know that problem when working with displacement tesellation and that fixed it. In this scenario probably works but if you dont want to try.
