Was trying to use custom stencil to achieve multiple colored outline for in game actors following this tutorial:
The outline effect itself works perfectly, which would imply the stencil worked, however when I tried to lerp based on the stencil value itself, it seems the custom stencil node would always return 0 regardless of the set value(I attempted 1, 2, 3, 100, 255). I was expecting stencil 0 to turn off outline, 1 to have red outline, 2 for green, 3 or above for cyan, while regardless of the value I set, the outline would always be red…
The project setting was set in the following manner:
Yes I have figured out the why. It turns out I incorrectly hooked the base red node so it always overrides the other stencil colors. The engine itself is fine. This can be varified by creating an empty project, enable custom stencils in project settings, and notice everything works as expected. A rule of thumb is that if you turn on “stencil view” on the top left part of the game window and your object shows up with the correct stencil number, then it is working fine. Please, double check your shader.
As you can see, the surfaces have a color rendered to them (picked with the stencil index). So what I was taking from this is that, most likely, the CustomStencil is taken per pixel, so as the stencil map doesn’t actually have a stencil at the location of the outline, it is always 0. This would mean that the stencil information has to be moved too (like how we did with the outline calculation) and added to the stencil node’s UV input. Reworking the whole thing to use the already moved SceneTexture nodes’ colors and changing them to use stencil instead of depth left me with this:
You’ll have to make this for the other directions too and max up all of them for the final subtraction. The resulting mask can be used as the index for your color selection and, clamped to 0-1, as the alpha for the final lerp.
It’ll bring you something like this:
which is pretty much what we want, but this solution has unwanted issues when objects overlay:
But I think I might be able to solve this. Maybe some combined approach will do the trick, keeping the depth
All this leaves me with the question how the tutorials could work the way they were, and I think the only answer would be, that the older engine version was calculating the stencil information differently. There is a hint over at [ParallelCube][3] that by v4.19 there’s some changes concerning pixel position calculation, but I didn’t find anything else about it.
Cheers and good luck!
PS: just found that there’s a github link at [ParallelCube][3] with up to date uassets
It’ll render something like this when overlapping:
I guess I’ll be working with this from now on and adapt it when needed.
PPS: You can see in his code, that he actually uses the Stencil information instead of the depth from the start (so very similar to what I did) which is why he can color the outlines in the end. The video tutorial you linked uses depth nodes (at least he did in the other video on his page) and it shouldn’t actually work. Either it’s some version differences, or he actually renders an inline rather than an outline (which would still have the stencil information). I stumbled over something similar when I was trying things out.
