For that you will need to use floatHDR format. But depending on the world size of the heightmap you may still experience some light stepping artifacts due to GPU precision.
I actually used the same technique for the kite demo so that any meshes on the terrain had a grass transition before they hit the terrain. This ended up only working if the gradient was fairly large (~1 meter or so with modulation using heightmaps to sharpen) since the map had such a huge Z range. The blend modulation helped to break up the artifacts but there were still places where it didn’t quite work as expected.
I also used to so that lakes/rivers automatically got a rock edge where they intersected with the terrain. In that case the artifacts were much less noticeable because I used the rock/pebble heightmap to break it up and that pattern was so high frequency that it dominated any quantization artifacts.
technically you could probably get by with less than HDR if your landscape has a fairly constrained z range. lets say your z range is 255 meters… with 8 bit you will be able to represent ‘terraces’ 1 meter high and nothing more.