on the topic of blending meshes with the landscape, I wonder why the approach of using decals is never really mentioned or considered
quick and dirty test:
Mainly because of draw call inflation. It is generally better, than vertex painting, as you can have a dedicated decal for each layer but prone to same limitations overall. But it is indeed path less traveled and not everyone is aware of it.
well sure, as with everything there’s a tradeoff.
vertex painting can be made scalable (turn off the feature via staticswitch in a material instance applied to the mesh LOD), but the fancy material blending involves duplicating the shader setup of the landscape layer into the mesh’s shader. and blending over multiple landscape layers is unthinkable
at least the decal method doesn’t over-complicate the other materials, and can be made broken down into pieces to blend against different landscape layers. it’s also scalable in its own way (decal fade screen size).
overall with the decal method I find the limitations less… limiting
For non instanced meshes you can precalculate per vertex world position offset and store it as vertex color. This way there is no need for any texture sampling at vertex shader or even need for heightmaps. This would need some amount of custom code. Problem is that each instance would need unique vertex color stream wich cause some memory overhead.
Well, yeah, I’d agree. Decals are more sustainable system. Especially if you can place them procedurally over the meshes.
Soft depth testing on the other hand is a universal method of blending meshes into terrain that fully scalable for N meshes and N terrain layers.
It would have been my method of choice, if not combination of parallax I mentioned earlier, and Incorrect velocities in blend region.
Up to this point, I’m confident, that from selection of known methods, tapping into terrain texture arrays/blend maps or virtual textures, to blend meshes over terrain, is number one all round, closely trailed by old good mesh skirts. First is being out of option in stock UE4. Latter is fine and doable, but still suffers from parallax and velocity issues.
That leaves us with dithered blend for meshes(which is also introducing unnecessary parallax), where dithered region is controlled by sampling terrain’s heightfield or other means, conventional vertex painting terrain layers on top of meshes and decals.
May I ask how did you do soft depth testing in ue4?
was going to ask the same. a quick search only showed that apparently that’s how Cryengine calls it (link) but there it’s described as having the mesh getting “integrated” into the terrain
from what I can see, in Unreal you’d need to either use translucent for the rocks or a dither pattern with temporal AA to achieve the softness. or did I miss something?
that ground texture looks familiar btw :rolleyes:
Ground texture is a demo megascans one.
Not being a CryEngine user, I was initially pointed to this technique, demonstrated in a video, by ZeroNight.
Whatever you call it, in the essence, it is just alpha blended terrain. Not sure how it is done in CryEngine, but in UE4 my workflow was as follows:
- Run early depth pass for terrain conventionally.
- For base pass of terrain, ensure that terrain is the last thing to be rendered in base pass from solid and masked objects
- switch depth test to always
- configure blend state to use source alpha
- offset the terrain closer to the camera by amount, equal to maximum soft depth test rage plus small bias
- calculate final alpha as a typical depth fade, multiplied by manual depth test, biased by a range of soft depth blend.
That is only doable after quite a few modifications to the engine though.
In CE, soft depth test and integrating meshes into terrain are two separate independent things, but latter greatly reinforces the first one.
Integrating meshes into terrain allows you to have minimal soft depth test range, while giving you large blend regions.
Without modifying terrain to closely follow the mesh, You would need to ramp up soft depth blend window so high, that you will start seeing a ton of problems, ranging from misplaced shadows to dreaded blurring in the blend region, caused by the fact, that velocities will be calculated using original terrain depth before offset.
And even at soft test range of just few units, you will already feel drift between surfaces being blended, so this is the effect, not suitable for everyone. I personally dislike it.
It is not free in terms of performance either. You are running depth pre-pass, but not getting any benefit from it and paying full overdraw hit from landscape(This is specific to my implementation at least).
Last but not least, You can’t selectively blend things. The terrain will be eating feet of your characters alive >< .
If you can get your mesh to follow terrain quite close(pulling up vertices of tree roots using heightmaps is exactly the case) soft depth test is good.
In other cases, it is next to useless, and decals, suggested by above, are actually a better alternative, as I see it.
Many thanks for the explanation.
@**, **Not quite sure what you’re doing with decals there, placing decals manualy around a mesh?
thanks for the breakdown Deathrey 
indeed I had the impression you’d need to modify the engine for that (sadly means most of us won’t get to try it)
and yes that texture is a demo from megascans, it’s the same texture that’s visible in my decal example above 
@Maximum-Dev the process is quite simple to already get to the point I showed.
I have a decal the size of the mesh. I put it just slightly on top of the landscape centered at the mesh and so it just naturally fades out vertically.
then the decal material has the same textures/params as the landscape, except tessellation is off since it doesnt work on decals (but it naturally projects on top of the landscape’s own tessellation)
the one thing that’s important here is that both the landscape and the decal have the textures’ UVs coming from the worldposition XY, and that’s why the landscape and the decals match 100% - btw this causes landscape displacement to only work “upwards” rather than “towards the normal” but there’s a workaround
after having the proof of concept working I made a blueprint that will populate the decal material’s params from the landscape’s own params based on a hit trace result. so now I can move the decal around and it will automatically change the material to match the landscape layer underneath.
of course for that to work you need to have landscape layers with physical materials assigned (if your AL stuff is any indication, you won’t be able to do it like that)
the above test was done placing a single decal manually. after that I also made a blueprint that will place multiple decals around the mesh (based also on traces to get something of a silhouette of the mesh, and the correct placement in relation to the landscape), so I can already have multiple decals adapting my mesh to different landscape layers automatically
at this point the ‘multiple decals’ thing is quite rough though. the amount of decals don’t quite conform to the shape of the mesh which leads to many unnecessary decals.
at this point I’m struggling between “automatic placement” and “efficient placement” for my decals, specially regarding drawcalls. in the perfect world the system would determine if there’d only be need for one layer underneath (and so just use one decal), or determine and conform a minimal amount of decals (basically one per underlying layer). just not quite there yet
all in all I find this method more simple and efficient workflow-wise, while performance-wise I think it’s just a legit tradeoff (have to keep drawcalls under control).
I did fail to notice that.
Anyhow, elaborating on blending with decals, Is draw the only real showstopper?
How about drawing those decals as instanced static meshes instead of decal actors(I believe you still need some changes to get instance position in decal material, but I haven’t looked at it)
so far these are the limitations I’ve found:
- decals won’t have the landscape tessellation so you might notice that when the decal reaches the rock mesh, the landscape texture projected on the mesh no longer has the tessellation and instead just matches the mesh topology. but unless you use some crazy high displacement it’s not really visible (in my screenshot above it’s really not evident)
- another small limitation is that the decal rotation ‘contributes’ to the normalmap, so the more uneven your landscape is, the more spots you’ll have where the decal lighting can feel a little off. but since I fade the edges of the decal anyway, there will never be a hard transition so it’s quite alright
- since I rely on WorldPosition-XY as UVs, if the decal gets projected on a vertical surface it will stretch really badly. decals don’t seem to have any angle filtering options, and the decal material is limited to either reading or writing the world normals (but not both) so it’s not trivial to reduce the opacity based on the projected angle (reconstructing the normals from depth might work but it’s not accurate and adds to the cost anyway). this for me is the biggest issue at this point. I could stop using WorldPosition-XY as UVs but then the magic of having the landscape texture match the decal 100% would be gone
about drawing the decals as instanced static meshes, I fail to understand what you mean
the rocks/meshes can indeed be added as foliage to reduce drawcalls (but that’s irrelevant to the blending we’re discussing). unless you mean using Mesh Decals somehow? (but they are terrible)
We can live with that.
Normal projection for decal is dependent on decal rotation. Suggest skipping tangent space normals, and outputting world space normals by rotating result of normal map sample based on difference between world up and decal up vectors.
Well, yes.
Indeed, you can use partial derivative of underlying position to estimate normal, that you would use to blend between projections.
Generally, I don’t agree here, that it is game breaker. It most cases it is safe to leave it as it is.
But this issue is tied together with another one. You really need those underlying normals anyway to get rid of lit backsides of the mesh, where the decal rotated normals to point towards the light, while it should be fully shaded.
I will revert with clarification a bit later.
this crossed my mind before, just didn’t have time to try it yet
however it will still cause wrong lighting on uneven terrain since the decal only has one up vector so it will only match the shading at one specific point but not all of the surface. as such, it’s unclear to me at this point what would be the benefit of this
that’s what I hinted at before. I’m still not convinced it will be good enough since partial derivative reconstruction of normals usually has some shimmering pixels. I’ll have to find out!
depends on the case. in my example above, on the back of that rock there was already a very vertical “wall” and it looked really bad (I just didnt show it!). in such cases a hard cut (i.e. not having the feature) would even be more natural.
but the example above was a rock on a flat slope. now imagine a rock on a steep slope, keeping in mind that the UVs are top-down. it can cause some bad stretching on a lot of cases.
I’ll show some pictures later to illustrate the issue
I also fail to understand this. if the decal is rotated towards the light (but projected on the backside of the mesh), isn’t the decal in shadows already? also don’t decals get light properly anyway? (i.e. if you put a decal on flat terrain and rotate it upwards the terrain becomes dark). I guess this means some more testing from my side to catch this possible pitfall (so far I only tested with ‘around noon time’ light setup)
alright thanks 
It is not a question of benefit or not. You need it to match decal normal map with terrain layers normal maps, otherwise normal details will be pointing in the offing. Unless you want to re-construct terrain normal per pixel in decal shader, there is no better alternative to using decal orientation to slant normal map. And the decal would be always perpendicular to terrain or half way between terrain and rock, giving you average terrain blend normal.
Most likely yep, that is a dead end. Second alternative is a normal pre-pass. But no. Just no. Isn’t happening.
Yep, that bad, but in a general case, I’d accept this issue gladly, should it be the only one.
Decal doesn’t have normals or lighting. It just blends its normal map on top of existing world normal. During the blend it is possible to push normals, that facing away from the light, towards the light. It will look as a rock with SSS.
This is what I meant under decals as ISMs:
https://i.gyazo.com/8e57308bf26a9124d4761a031520fc77.gif
right. placing the decal to match the normal perpendicular to the landscape will already give a ‘correct enough’ result (again, will be slightly off if the terrain is uneven - but it’s good enough and better than reconstructing the terrain normal)
it blends its normalmap on top of the existing world normal, but it uses the decal rotation to ‘rotate’ the normals. this ties in to the point above of needing to place it parallel to the terrain. as such, I still don’t get the mesh backface issue you described, but I’ll just have to test to see it
I don’t understand what I’m seeing here 
you’re painting foliage using a mesh… and that’s as far as I can tell for sure. from that I can only speculate that your mesh uses a mesh decal shader, but then somehow it properly gets each layer below? is this vanilla UE4 or with the changes you mentioned before?
but if you’re using mesh decals… please don’t get me started on how bad they are
Yeeees, it is a mesh decal, so about time to get started.
Pardon me, what is so bad about them? I mean technically, what is the difference between decal actor and mesh decal ? I don’t see any fundamental disparities to be fair.
so this is what I mean with the rock slope stretching
to me it ruins the effect 
and the issue is that if any automated method is used for placing the decals, it’s hard to prevent it from happening. so I’d really prefer to find a solution within the shader
fine, let’s get started
mesh decals aren’t really decals. they keep their own topology and don’t project onto the geometry underneath. figuring out the math for the clipping and projection on an arbitrary mesh is very non-trivial (someone tried it at work at some point) compared to just using a cube as regular decals do. therefore mesh decals pretty much just act as translucent meshes with the benefits of being rendered in the decal pass.
since they don’t really project onto the geometry they are prone to issues. first off there’s parallaxing. the more difference there is between the projected geometry and the mesh decal geometry the more parallax you get (and good luck matching arbitrary landscape->mesh topology)
there’s also clipping/z-fighting, which have some hacky hardcoded “solution” which only causes different issues. have a read here: https://forums.unrealengine.com/deve…stance-problem
edit: the “floating decals towards the camera” issue is something I can’t reproduce in 4.19 atm. but instead I now I get z-fighting at high distances. this is something that can be lived with (since you’d fade them out on distance anyway). but the parallaxing I just can’t (to me it behaves like a floating mesh rather than a projected texture)
in any case mesh decals require DBuffer decals which I disabled to get rid of the pre-pass to get any sort of decent landscape+tessellation performance
Mesh decals are heavier, hence the shader is a bit bulkier, Unlike decal actors, they have a primitive uniform buffer and they are sorted differently.
Besides that, I am afraid, differences conclude.
In our context:
Decal is deferred, not dBuffer decal.
Decal mesh is just a low poly sphere. Could use cube, whatever.
Nothing is floating around. Proper projection is done.
It is a bit strange, that you assumed, I would place floating planes in the mid of nowhere and call it a blend.
https://i.gyazo.com/01780a937e5670a4628e6c399f31245a.jpg
https://i.gyazo.com/186a9a765b147da7ce93f6cb11d032f8.jpg