Bottom line up front: I’m trying to use the Sobel edge post process effect in DX11. It has this strange behavior.
When a mesh or masked sprite is in front of another mesh or masked sprite or BSP with default material, the outline gets drawn with the correct thickness and color. But when the mesh or sprite gets drawn in front of an unlit BSP (emissive = 0) or in front of the plain black world background, the outline gets drawn really thick and in white. I’m wondering if there’s a way to fix that.
And now the background: There’s this longstanding problem I’m trying to work around.
I’m trying to use the Sobel edge post process filter with translucent decals. It doesn’t work because UE3’s DX9 renderer writes depth with translucent decals.
In this picture you see a translucent decal on the ground. The colorful square shows a material that represents pixel depth as a color. The translucent decal should be invisible behind that square (and it is invisible in DX11), but you can see that in DX9, depth is getting written. That messes up the Sobel edge, which wants to draw outlines in the middle of the decal because of those depth disparities.
So if I want to have both the Sobel edge filter and translucent decals, then I need to fix the DX9 renderer or switch to DX11. And if I want to use the Sobel edge filter, then I need to fix the thick-white-outline problem displayed at top, or I need to switch to DX9. I don’t really want to try working on Epic’s DX9 renderer. I’m now exploring DX11. And if DX11 just will not work with my filter, then I either need to leave out the Sobel edge filter or I need to go back to DX9 and get rid of all translucent decals.