I’m trying to make an outline Post-Process material where the default color is black and be able to change the color based on a CustomDepth Stencil Value after searching around I found a working solution but it has an annoying problem!
When 2 meshes overlap I get issues like this:
As you can see the outlines of the back meshes are visible which is not what I want!
What I really want is for the outline of the mesh in front to appear where both meshes overlap!
I also have this problem where the outline renders behind a wall through the wall’s outline:
this problem seems to be the result of how I’m generating the alpha to lerp between the PostProcessInput0 and the Outline color where I add the bit masks together and then multiply the result with the outline logic:
Any help is very appreciated!
EDIT: for the seeing-through wall problem all I had to do is to enable Render CustomDepth Pass on every mesh now i only have the overlap problem
EDIT2: After further debugging I discovered that the issue is actually from the outline logic it self! I’m using Depth and normal to create the outline using hlsl and the normal part of the logic seems to be the issue:
as you can see the outline on the Sphere has thickness variation caused by the normal side of a the Laplacian filter similar to this approach https://www.youtube.com/watch?v=Ptuw9mxekh0