Help with outline on objects that have custom depth enabled

[FrancescoDeso]

I’m making an outline effect with multiple colors using stencil. In order to make it I looked around a lot on forums, youtube and so on. But there is still one thing that I can’t solve, and it’s outlines on objects that have custom depth enabled.

The main idea is to sample each pixel around the current pixel and if one of them doesn’t have a custom depth then I draw the outline. The problem is that it doesn’t take into account other objects that have custom depth, so the result looks like this:

In this image each object that has an outline has a different stencil index, while all other objects in the scene don’t have custom depth enabled. As you can see the outlines “stop” when the objects with custom depth overlap, which is what I don’t want. I would like in this case that the pyramid outline was fully shown (because it’s on top of the cube and in front of the pillar in the back).

To fix this I tried to do this: for any pixel around the current one I check if it has custom depth enabled and if the stencil index is different from the one on the current pixel. If so, I consider that pixel as if it didn’t have custom depth, so I draw the outline. This kinda works, this is the result:

As you can see now there are “too many” outlines. I can’t find a way to fix this at all… any ideas?

[FrancescoDeso]

I seem to have solved it:

To solve it I do this: for any neighbour pixel I take the difference between the custom depth of the current pixel and the custom depth of the neighbour pixel; if the result is greater or equal to a given threshold then I don’t draw the outline, otherwise I draw it.
Then I check if the result neighbour pixel has the same stencil value of the current pixel; if so, I ignore the result, otherwise it will completely fill the object instead of drawing the outline.

The threshold musut be a low value. I’m using something like 0.001, but it still has some problems when objects with different stencils overlap, like this:

I have no idea how to fix this.

Also, if different objects have the same stencil it doesn’t draw the outline, like this:

I would like that different objects, even though they might have the same stencil, had an outline around each one. But I don’t know how to do that either… how could I distinguish between differet objects knowing only the depth information?

It’s ok right know, but I would like to fix these problems… I’m out of ideas.

[Darrel_Leigh]

I’m curious how you did this?