I wrote a shader of my own ten years ago that does something similar.
The idea is to include an alpha map in each material, and add the alpha map to the blend value between terrain layers.
The grass would then have the alpha map opaque where it has “blades.”
The blending function would then make it so that:
- when the blend is 100% grass, only include the grass texture
- when the blend is 0% grass, only include the mud texture
- when the blend is 50%, lerp between grass at 100% alpha and mud at 0% alpha
This turns out to be a single lerp based on a scaled sum of the layer blend function, and the texture alpha
The main draw-back was that it’s more work to create good alpha materials that blend well together using this method.
I believe the same kind of material could be created in the Unreal terrain material system, although you’d probably only get 3 layers out of it if you want the fancy PBR material properties.