Thanks for the suggestion to check out the vertices, that was helpful. It seems my problem is due to how the vertices are connected as squares, which a single line between the North-West and South-East corners. So when the cliff runs North-East to South-West, it runs against the connecting lines and causes a jagged cliff-face.
Do you know of any way to change the way the terrain vertices are connected? I’m guessing this is something that shouldn’t be messed with…
About photoshop, I actually built the whole landscape in photoshop (it does support 16 bit), so there’s no data loss. I have found photoshop to be much better than tools like World Machine or World Creator for what I wanted to do, which involved some very specific sculpting along with the ability to procedurally modify layers and transitions as needed.
It’s also really nice because the same painting, with a few blending functions, works perfectly as a map of the world. So I can draw my cities and roads, and then shape my terrain around those as well.