Outline effect on mesh intersection. Is it possible?

Based on Tom Looman’s material I came up with my own post-process effect for outlines and the result is quite good:


But it doesn’t work when I have 2 meshes that use custom depth overlaping each other:


So, is it possible to have an outline right in their intersection?

I’m guessing I could use stencils but then I wouldn’t I need one stencil for each mesh!? I’m already using stencil for different outline colors btw, so I don’t think that’s an option for me.


I think in case you use depth field you can generate such effect directly in objects materials using “distance to nearest surface”

If you are already using different stencil values for the different colors, you should be able to use the same mask to make the outlines work across eachother. You will just have to compare stencil values as well as compare custom depth. You may have to enforce and order or priority so that one of the colors wins in that case.

I should explain it better, I’m using stencil for either have a black outline (those are interactive objects in the game) or have an outline using a slightly darker color of the texture, so those spheres are using the same stencil.

That gave me an idea though, I guess I can use more stencils for different functions but won’t that be costly? Will weaker hardware be able to handle that?

It all depends on how you end up doing it.

Now that you mention it, I do notice some of the lighting being picked up in the outline.

If you don’t have tons of unique colors, you could do essentially the same thing you are doing now, but using some pre-determined ranges of the stencil values rather than specific numbers. The math should not end up being so different.

For example, you could say values 0-50 are for shades of colors. Then Some specific value is for static environment or black outlines.

So you would first check for black, then check for differences within the ranges, if that makes sense.

Updating this after long time, after trying a lot of things including different stencils, I found this post (in japanese which I don’t speak):
It combines Custom Depth with Normals to give more details to the edges and it works but there are still a few tweaks I would like to make.
Here you can see the boxes intersecting each other.

I’m still having 2 problems: The first is that in some cases this just doesn’t work. It seems to me that the outline created by the custom depth is being drawn inside the mesh while the normals are being drawn outside which gives 2 different outline thickness sometimes. Also, from some angles it just doesn’t work.

The other problem is that if I have an object with no custom depth in front of one with, I sometimes get an outline on it too:

The plants have no Custom Depth but the wall does.

The method on the blog uses the dot product of the pixel normals with a constant vector:

Playing with that constant I noticed that I can change the behavior of the shader to be more like I want but I can’t find a “perfect” value for it (don’t even know if it’s possible) and I’m not sure of where this constant is coming from.

I’m kinda stuck here, any suggestions would be appreciated.