I use this trick in my custom pathfinder too. Maybe once I will get in a mood to make it with navmesh… Earlier I made some tests, if the units have a target approx 5m far (probably can be increased), and the navmesh is dense enough, they keep the formation fine.
moreover, the problems you can face when using navmesh for RTS:
- handling different unit sizes: in theory it requires separate navmeshes to avoid large units getting stuck or colliding too much
- a workaround: due to my tests, in Age of Empires 3 all the units had a similar infantry sized collision, but tried to keep distance afterwards
- handling different capabilities: if different types of units can pass different areas (e.g. a tank can break through small walls or vegetation), in theory it would also require separate navmeshes with different settings
- a workaround: Rama made navmesh movement filtering blueprints (you can find it in the forum), it requires to place invisible navmesh modifier blocks (probably it can be attached to actors as component/volume)
- ship movement: I don’t know how to set up navmesh for them
- you are lucky if don’t use ships
