I have encountered the same problem. Considering the default landscape you get converts to nanite just fine, i also assume it has something to do with the Z-scaling. If only there was a way we could apply the transform just like we do in blender to reset all the scales to 1 without changing the actual size.
I did try exporting the heightmap over to blender to try and scale it there and applying the transformation, but with a map my size, blender would straight up take all my 64 GB of ram before crashing. It is worth a try if your heightmap is smaller than 4k by 4k for example.