Sound like you should have picked something a little easier to start with if you have no clue on what is wrong at all.
First of. I’ll review your formula in a sec. But of the top of my head you have an extra division by 2pi. I dont remember that being there 100%, but I also only remember the math about 20% so that’s not like saying much.
Second, forget the texture samples for the normal. The gerstner formula already gives you the masks. (Or you could even use ddy/ddx).
3rd, I cant read the divide values on the clusters. Make sure it matches the number of waves 1 to 1. 8 waves means you need to divide the result by 8 before output.
Likewise, adding the 2 clusters of 8 waves for the output means you need to divide by 2.
4th. In the image world displacement is not plugged in. The result of the divide needs to be fed into it.
Tessellation multiplier can be either 0 or 1 or more. It depends on the mesh you assign the material to, not on the material itself.
The mesh. I use a custom plane of 100m by 100m that has 8 LODs with varying degrees of vertices.
I use 250 for LOD0, 100 for LOD2.
With the tessellation multiplier of 2 you therefore get
500x500 vertices at LOD0, 400x400 at LOd1, 300, 200,100, 50,25.
The Usefulness of the multiplier is that you can reduce or increase geometry on the fly to gain some performance.
Forget the colors and everything else. Enable tessellation as PN triangles, plug it into a non translucent material, let it compile. Apply it to the mesh and test.
If it works fine you can move on to the normals. Once those work fine you then move on to coloring. After coloring you move on to enabling translucency (so you dont wait 2 days in front of the screen for shaders to compile).
Water as a fixed index of refraction (of 1.33 I think? It’s in the unreal docs). When you need to set the specular to the PBR default value as well. Also in the ue4 docs.
Likewise metallic is usually 0 (even if some unreal technical artists like to ramp the metallic value up a bit).
Roughness is generally either at 0, or it needs to be derived from the normal map by masking the Z (b channel) so that the tip of the wave is non reflective while the lower part reflects.
You may wish to multiply that mask (and clamp it 0 to 1) to increase or decrease the effect at will.
When you turn on transparency you’ll get wierd artifacts from the back faces of the wave showing before the front faces.
Either do not turn on transparency, or (if that’s not possible as it shouldn’t be in any somewhat realistic/non toon shaded project) you need to work around it by rendering the material in separate passes and filtering what appears on screen within a limited mask.
The community ocean project has some custom nodes to manage this I believe.
I ended up using the standard way of preventing outlines of the inside of a mesh from showing in my project and that works fine too.
Your parameters on the gerstner waves /instances material could be causing the displacement to be super tall and out of your field of view (as you describe the issue).
So, it may be working, you might just need to reduce the steepness to be able to see it.