Huge Gaps in Wall due to Tessellation

I am using a displacement map to create tessellation on a static mesh. Whenever I do this, I get cracks in the corners of the walls. The higher the tessellation Strength, the larger the gaps.



If I check the crack free displacement box, it appears to fix the problem.


However, when I try to apply the material to my mesh, the shaders wont compile unless I disable tessellation completely.


This also happens If I try to use PN triangles instead of flat tessellation. In addition, adaptive tessellation creates terrible results.
Here is how I set up the tessellation. The displacement map is in the red channel of my roughness map.
Its attached below, sorry it wont let me insert the picture here.


Is there anything I can do to be able to fix this?
Any help would be appreciated.

Thank you in advance.

PS. Formatting posts on this forum so it looks nice is a pain in the ■■■.