Nanite is not an alternative, it’s completely orthogonal to 99% adequate use cases of tessellation.
Currently, it’s not possible to do tessellation tricks anymore in UE5 until someone returns it with custom engine build.
Vertex animation using low poly mesh, possibly with translucent material, or with procedurally generated/updated mesh (nanite doesn’t support any of these features)
Procedural displacements on meshes
Infinitely detailed ocean using 1 quad (but that’s probably not the best way)
The overwhelming majority of people were using tessellation with a texture in order to add additional static detail to static meshes. I’m not sure you would consider that an “adequate” use case, but it is what most people used it for. For that, Nanite produces far superior quality at higher performance, at the cost of being somewhat less convenient depending on how you generated/acquired your content.
The second most common thing people used it for was as you’ve stated above: Procedural animation/deformation. But primarily people were doing that specifically for two things: Water surfaces and landscape deformation (such as snow). Both have better replacements coming in UE5.
For everything else, there’s still WPO. You’ll just have to pre-tessellate them and use LODs. Perhaps not ideal, but it is still usable.
By design, it generates extra vertex between LODs whivh is prone to causing gaps.
You would then have to try and adjust so that the gaps wouldn’t exist.
Performance wise, the best possible system is to replace in quads like you would with an octree or more properly quadtree setup.
It’s complex to get the aggregation just right, and it also mostly doesn’t matter past 10m from view unless you are doing crazy stuff with the waves.
You could potentially set a single mesh with normal maps on it.
And then overly a set of 10x10 LOD meshes that moves with the character to give the effect of the wave body just around the player. (Well that’s too much too being 100 draw calls wasted. Maybe 5x5 would be best…)
It may actually end up outperforming even the aggregation methods…
At least with default unreal, you can’t tesselate the mesh at runtime to simulate water like that.
The best approaches are the nvidia flex implementations for things like that.
And even if you could, the tessellation is mostly global and not analytical (where needed like splashes)
In a game you can fake the waves with flow maps and analytical splashes.
It looks decent even if nowhere near as good as either of the 2 methods.
Mostly, you can’t really rely on a shader to do a simulation since the calcs are much more complex.
I think nvidia ended up using spheres to produce and offset the sheet of water with a somewhat accurate result (which can optionally be set to be on cpu).
Anyway, if that’s what you need, look into waveworks.
The idea is to reduce the overall tris count of distant LODs without compromising quality or creating seams during transitions.
You create the lower level LOD first, then you add more and more divisions intelligently so that your edges of ALL lod(s) have the same number of vertices.
You can see all the LODs in the shared clip. They aren’t necessarily the best possible break-down, it’s all variations on a theme anyway.
Provided the edge vertex count is the same, your lowest LOD will have:
edge vert count * 4 tris.
For example, a square with 2 subdivisions, 4 vert on edge, will have 16 tris.
You balance an overall number of tris out with the rendering budget and you get decent to great results without any issues.
On top of having something that will work on any engine regardless.
As sizing goes, I suggest 10m^3 tiles with 30 sub divisions for a total of 120tris each tile.
OBVIOUSLY you have to use instancing, if you create non-instanced assets the rendering pipeline will get shredded by draw calls.
On low end systems make the last LOD a single vertex. Then replace the ocean out after something close to 50m to a solid slice with texturing. You shouldn’t be able to see any difference.