16bit PNG is a fine format, if exported properly, and will get rid of the quantization artifacts (stairstepping/terracing) when exported properly (from WorldMachine for example), if your issue is caused by lack of precision.
In case it’s not lack of precision, and the heightmap is supposed to look like this elevation-wise (not detail-wise), then the most common solution for improving landscape detail is using smaller scale when creating landscape, so you get better precision then one square per square meter.