Currently the SceneTexture:CustomStencil (thanks for adding that BTW!) can only be read in postprocess materials, at least as far as I could tell. I can only get errors out of it otherwise, even with Masked blend mode materials, which is where I want to use it.
I’m aware there might be pipeline stall issues with this, but in my specific case, I’d be quite happy with the stencil buffer from a frame or two ago. The problem I’m trying to solve is demonstrated in the attached image. Both green and red objects have the same geometry. The green object is entirely above the blue plane. The red object is half sunk down. I want the blue material to cover the outside of the red object, but NOT to cover the inside - much like a cup or boat floating in water, where the cup produces a cup-shaped hole in the water. I can use a different material for the outside and inside of the object.
The stencil buffer seems ideal for this, but I can’t use it outside post-processing. I can’t use pixel depth offsets either, because that will yank the inside of the object in front of the outside when it shouldn’t be. I also can’t do it with geometry due to the complexity and movement of the objects.
If there’s any other way to go about this, I’d love to know! So far everything I’ve tried has come up short. It seems like the stencil buffer is meant for this kind of thing, but I can’t use it in a masked or opaque shader :-(.
Thanks!
For what it’s worth, I am getting closer to achieving this effect, though not in a practical manner. I discovered that only Opaque & Masked materials are unable to read scene textures (I imagine because they are done in an earlier pass before stencil is available).
I moved the blue material to Translucent, and created a separate UE object for the inside and outside of the box. That, plus reading the stencil buffer produces this, which is what I’m going for, but with some big caveats:
Caveat one: If anyone from Epic is reading, it would be super great to have finer control over the stencil value! Right now it’s per-object, which is a big hammer. That means I have to spit my object in two, to have enough control over the stencil value, which is mighty inconvenient. I’d like to see per material at minimum, or ideally even per pixel from a material output. (It’s per pixel in the hardware).
Caveat two: Try as I might, I can’t get the translucent lighting model to behave anything like the opaque one, even with opacity=1. On my actual material it looks really bad, no matter the value of Translucency/Lighting Mode. Possibly I’m doing something wrong, but I don’t know what.
It’s really close though! It’s great that the stencil buffer became usable in 4.9. With a few more tweaks, it would do just what I want. An option to allow it to work with Masked materials, plus either per-pixel or at least per-material control, would be wonderful.