From experience:
don’t rely on 1 mesh with many vertices to get displacement going. This will bog down both the GPU and the CPU causing a bottleneck due to the face occlusion cannot work on a large mesh that’s always seen.
Basically anything over 1000x1000 verts immediately halves your FPS.
Even at 640x640 you’ll bottleneck your GPU and that’s what you are noticing with performance.
Instead, if you want to do stuff your self, make a smaller mesh of maybe around 50m, tile them all together with snapping, and apply a material that uses the same displacement map with World coordinates.
World Position - Mask R,G - Divide Map Size, Multiply .5
OR
World Position - Mask R,G - Divide Half Map Size. You can get the map size from selecting the landscape, and using the X or Y location (including - or +). they should be the same number.
On top of it, like the landscape you should consider at least 5 levels of LODs on the plan mesh. so that you can adjust the number of triangles based on distance.
My 50m mesh has 150x150 vertex at LOD0, 100x100 at LOD1, 50x50 (1:1) at LOD2, 30x30 at LOD3, 25x25(.5:1) at LOD4, and 10x10 at LOD5
This does help reduce the GPU load on constant distortion or formulas.
Another thing, the Tessellation Multiplier.
What that does is add geometry. so with a Tess multiplier of 2 my initial mesh becomes 300x300 vertices, and so on. This obviously starts to bottleneck things again if you have too many tris on screen.
Last but not least, there are some different techniques, it depends on what you are doing and what you need.
look into Alembic / Geometry Cache
and Morph Targets.
In both cases these use the same baseline GPU side calculation to distort the meshes, but the first one is much faster and a static mesh, the second one a skeletal mesh with an empty skeleton.
Hope that helps,
Best of luck.