The vram is only used when you display the landscape/keep it in memory.
In ue4 it used to be enough to close the minimap and let it process to load up around 120km^2 on a 1080ti.
If the mini map is displayed, all landscape tiles shown have to be loaded into memory with their relative costs. Causing the overflow.
Obviously. You cannot load all the tiles at once or you will also crash the engine by running out of Vram. You have to find how many tiles your machine allows you to work with, to do the acrual woek/sculp/spline part.
Or dont use the engine, which is a much better alternative. Try houdini.