Buggy material behavior in DX11. Any workarounds?

Legacy UDK Content Creation and Design
1

Bottom line up front: I’m trying to use the Sobel edge post process effect in DX11. It has this strange behavior.

When a mesh or masked sprite is in front of another mesh or masked sprite or BSP with default material, the outline gets drawn with the correct thickness and color. But when the mesh or sprite gets drawn in front of an unlit BSP (emissive = 0) or in front of the plain black world background, the outline gets drawn really thick and in white. I’m wondering if there’s a way to fix that.

And now the background: There’s this longstanding problem I’m trying to work around.

https://forums.unrealengine.com/legacy-tools-unreal-engine-3-udk/udk-content-creation-and-design/1383547-has-anyone-ever-combined-sobel-edge-post-process-outline-effect-with-translucent-decals

I’m trying to use the Sobel edge post process filter with translucent decals. It doesn’t work because UE3’s DX9 renderer writes depth with translucent decals.

In this picture you see a translucent decal on the ground. The colorful square shows a material that represents pixel depth as a color. The translucent decal should be invisible behind that square (and it is invisible in DX11), but you can see that in DX9, depth is getting written. That messes up the Sobel edge, which wants to draw outlines in the middle of the decal because of those depth disparities.

So if I want to have both the Sobel edge filter and translucent decals, then I need to fix the DX9 renderer or switch to DX11. And if I want to use the Sobel edge filter, then I need to fix the thick-white-outline problem displayed at top, or I need to switch to DX9. I don’t really want to try working on Epic’s DX9 renderer. I’m now exploring DX11. And if DX11 just will not work with my filter, then I either need to leave out the Sobel edge filter or I need to go back to DX9 and get rid of all translucent decals.

2

is the thick line against the black background something that happens in editor or in game? the editor handles the void differently as the game so make sure to test it in the right target - IIRC one of the 2 handles it as having max depth and the ohter one just ignores it.
you might want to add a skybox (even if just black) anyway

if you can get around this I’m curious if you’re planning to ship your game as DX11-only though (seems DX9’s and DX11’s problems kinda make it mutually exclusive for you?). DX11 might run fine for you in UE3 but I can’t stress out enough how problematic it is on different setups. you might end up with a game that -for many people- could be broken beyond your reach

3

Thanks for taking a look Chosker. The thick white line is in both the editor and the game when running in DX11. It appears against both the void and against BSP with an unlit material when emissive = 0. I haven’t tried it with other materials, but maybe I should try it with a black static mesh sphere.

I run into problems either way: With DX9, I can’t have translucent decals with the Sobel edge effect. With DX11, I can’t have the Sobel effect in front of a black material.

I might have gone with DX11 if it didn’t have this problem, but it does, so I might have to stick with DX9. And anyway, maybe I should just accept that UE3’s DX11 is broken (and not keep trying to use it until my GPU explodes) and force myself to stay in DX9.