Material Layer Blending + Nanite Displacemen = Spiky Mess?

Curently trying to create a creek scene and I created a layered material. After fiddling for a few hours, I got the base rock material to look right with it’s displacement parameters. However, as soon as I add a new layer, the height of both of them seems to get overlapped. In this case, I’m adding a forest ground with some moss, twigs and grass. As you can see if I adjust the blending weight to either of the extremes, both of the materials behave as expected. But if there is any blending, I get stretched geo on any area which is not absolute 0 or 1 in the layer weight.
Do you know what I might be missing here? Thanks in advance!