I’d like to do the same thing as you. I suspect the solution would be similar to how the outline is drawn and you want to see if any of the outlined object’s pixel neighbors are occluded then you want to draw the outline. Hope this helps I’m off to figure it out myself.
I added in a parameter for the color so i can be changed. I’m planning on modifying the material at runtime to change it’s color and maybe make it pulse.
Here’s a screenshot where I set the alpha to 1 to make it more noticeable.
Conceptually, the difference between those two versions is that the first one applies the foreground mask as a final multiply of the sobel edge operation, while the second version applies the masking inside of the sobel operation at each step.
I’m sure there is a more efficient way of doing this to get the same result but this is what I got to work. If there is a more efficient method I’d like to see it.
One caveat of this is that if more than one object is in the custom depth and they overlap the intersection is not outlined, however my plans for now are that i would only have one object in the buffer at a time.
Is the object occulled in the stencil buffer? If not then it’s a lot like just using the custom depth to outline an object.
Also the custom stencil buffer isn’t available on all platform for instance Mac OS X OpenGL but is available in the metal renderer in 10.11.
I was unable to get it to work even after I upgraded and my macbook pro is new enough to run it.
I’d be interested in seeing a screenshot of your material in action.
Yeah you can outline an object without drawing the occulled parts by adding the custom depth/ scene depth check at the end of mine and not have to use the stencil. I had that working with that addition to Tom Loomans outline material.
But I wanted to draw the outline at the edge were it was also occulled and not just not have it draw, like what numsi550 wanted above.
Gald to be of service and contribute back for all the help i’ve gotten from tutorials. You post about not drawing the outline when occulled helped me figure that part out, which then led to this material.
Oh no, i HAVE TO use the stencil. We have a whole bunch of post process materials that need to have different effect on different meshes based on the mask. So i can have blue outline on meshes in stencil id 4 and yellow outline for id 5, for example. IDs 0-3 are used by another effect, and so on. So i’d say this one is better. Of course if it was the only thing i would use custom depth for, i wouldn’t bother with stencils.