Updating this after long time, after trying a lot of things including different stencils, I found this post (in japanese which I don’t speak): http://ogimafu.blogspot.com.br/2015/04/ue4.html
It combines Custom Depth with Normals to give more details to the edges and it works but there are still a few tweaks I would like to make.
Here you can see the boxes intersecting each other.
I’m still having 2 problems: The first is that in some cases this just doesn’t work. It seems to me that the outline created by the custom depth is being drawn inside the mesh while the normals are being drawn outside which gives 2 different outline thickness sometimes. Also, from some angles it just doesn’t work.
The other problem is that if I have an object with no custom depth in front of one with, I sometimes get an outline on it too:
The plants have no Custom Depth but the wall does.
The method on the blog uses the dot product of the pixel normals with a constant vector:
Playing with that constant I noticed that I can change the behavior of the shader to be more like I want but I can’t find a “perfect” value for it (don’t even know if it’s possible) and I’m not sure of where this constant is coming from.
I’m kinda stuck here, any suggestions would be appreciated.