RECEIVE_DECAL Stencil and Masked materials

Development Rendering
, ,
1

We are trying to make use of the ‘RECEIVE_DECAL’ stencil bit after base pass and have noticed that it doesn’t seem to respect masked materials.

We have a mesh that is set to not receive decals. When rendered with a masked material, the pixels that have been masked out and/or are dithered are still showing up in the stencil bit as not receiving decals.

Our expectation is that the pixels that get masked out would show as capable of receiving decals (assuming the background geo/pixels are able to receive decals).

Is this expected behavior? It seems like RayTracing has a workaround by running a separate pass to capture these masked/dithered pixels - “DitheredLODFadingOutMaskPass” - but we’d like to avoid having to add an entire new pass.

I have also noticed that this issue goes away if I toggle on “r.EarlyZPassOnlyMaterialMasking”, but we can’t afford to take our opaque geo out of the early depth pass, so is not a reasonable solution for us.

2

Hi there,

The issue about where masked-out pixels appear as “not receiving decals”,is expected within the UE render pipeline.

In this case, the mesh is marked as not receiving decals, so the Base Pass writes the stencil value accordingly. Masked pixels still inherit that stencil value, and the background geometry does not get an opportunity to contribute its own stencil bit.

For masked materials, opacity clipping is evaluated in the pixel shader. However, stencil state is configured at the draw-call level, and the GPU does not “undo” stencil writes for pixels that are later discarded by the opacity mask. As a result, even pixels that fail the opacity test may still retain the mesh’s stencil state. This happens because stencil writes occur at draw time, not after per-pixel masking is resolved.

When r.EarlyZPassOnlyMaterialMasking is enabled, masked materials are evaluated during the Early Z pass. Depth is written only for pixels where the mask passes, and the Base Pass runs only for those surviving pixels. In this configuration, stencil writes align correctly with masked depth, providing per-pixel accurate stencil for masked materials. However, as you noted, this forces masked materials into the Early Z pass and alters the depth prepass for opaque geometry.

With DitheredLODFadingOutMaskPass, ray tracing requires accurate visibility for masked pixels. To support this, UE runs a separate mask capture pass to evaluate dithering and store correct visibility in a buffer. This avoids incorrect stencil or depth results, but it does come with a performance cost.

As an alternative, using Custom Depth instead of stencil to control all meshes that receive decals may be a viable approach, depending on your project requirements.

I hope this clarifies the situation. Please let me know if you have any further questions.

Best regards,

Henry Liu

3

Hi,

​

UE5 has an Early Stencil stage, which is a performance optimization that leverages hierarchical stencil buffering to test and potentially discard pixels before the pixel shader runs. The stencil value(RECEIVE_DECAL) is set via pipeline state for each mesh draw call, not by the shader. That means the GPU treats it as fixed-function depth/stencil per draw-call behavior. The shader doesn’t explicitly control stencil.

The sequence effectively becomes:

→ Early stencil write (Replace with RECEIVE_DECAL)

→ Pixel shader executes

→ Opacity mask clip

By the time the clip happens, the stencil write has already happened.

​

What you said “if a pixel is discarded/clipped, it doesn’t write to the stencil buffer.” is closer to how the additional pass works for ray tracing visibility. That is the pass evaluates masked/dithered pixels and writes a visibility mask texture.

​

​

Cheers,

4

It’s my understanding if a pixel is discarded/clipped, it doesn’t write to the stencil buffer. Could you explain why that is not the case here?