Nanite is not an alternative, it’s completely orthogonal to 99% adequate use cases of tessellation.
Currently, it’s not possible to do tessellation tricks anymore in UE5 until someone returns it with custom engine build.
The overwhelming majority of people were using tessellation with a texture in order to add additional static detail to static meshes. I’m not sure you would consider that an “adequate” use case, but it is what most people used it for. For that, Nanite produces far superior quality at higher performance, at the cost of being somewhat less convenient depending on how you generated/acquired your content.
The second most common thing people used it for was as you’ve stated above: Procedural animation/deformation. But primarily people were doing that specifically for two things: Water surfaces and landscape deformation (such as snow). Both have better replacements coming in UE5.
For everything else, there’s still WPO. You’ll just have to pre-tessellate them and use LODs. Perhaps not ideal, but it is still usable.
By design, it generates extra vertex between LODs whivh is prone to causing gaps.
You would then have to try and adjust so that the gaps wouldn’t exist.
Performance wise, the best possible system is to replace in quads like you would with an octree or more properly quadtree setup.
It’s complex to get the aggregation just right, and it also mostly doesn’t matter past 10m from view unless you are doing crazy stuff with the waves.
You could potentially set a single mesh with normal maps on it.
And then overly a set of 10x10 LOD meshes that moves with the character to give the effect of the wave body just around the player. (Well that’s too much too being 100 draw calls wasted. Maybe 5x5 would be best…)
It may actually end up outperforming even the aggregation methods…
At least with default unreal, you can’t tesselate the mesh at runtime to simulate water like that.
The best approaches are the nvidia flex implementations for things like that.
And even if you could, the tessellation is mostly global and not analytical (where needed like splashes)
In a game you can fake the waves with flow maps and analytical splashes.
It looks decent even if nowhere near as good as either of the 2 methods.
Mostly, you can’t really rely on a shader to do a simulation since the calcs are much more complex.
I think nvidia ended up using spheres to produce and offset the sheet of water with a somewhat accurate result (which can optionally be set to be on cpu).
Anyway, if that’s what you need, look into waveworks.