Separate writing into CustomDepth & Stencil buffers

Development Rendering
, , , ,
1

I have a single mesh that I’d like to draw into the stencil buffer, without it affecting the CustomDepth buffer’s values.
So… I’d like to be able to specify a mesh to either draw into EITHER the CustomDepth buffer OR the Stencil buffer, but NOT necessarily both.

As far as I can tell, this isn’t possible currently. Could you give me any pointers on modifying the engine to accomplish this?

Thanks
-matt

2

Hi Matt,

If you take a look inside CustomDepthRendering.cpp you should find most of what you’re after. Objects that are flagged for custom depth are batched in a primitives list then written here in the DrawPrims function. There’s a block of code that overrides the depth test and stencil values:

if (bWriteCustomStencilValues)
{
	const uint32 CustomDepthStencilValue = PrimitiveSceneProxy->GetCustomDepthStencilValue();
	RHICmdList.SetDepthStencilState(TStaticDepthStencilState::GetRHI(), CustomDepthStencilValue);
}

I’d suggest starting by changing that data to override the depth and/or stencil states per object as needed, following how the existing flags and data are passed for reference such as the GetCustomDepthStencilValue above. If you’re drawing a lot of elements it may be of benefit to expand the Prims member of FCustomDepthPrimSet, either adding an additional sort or separating the lists if that’s better for your goal.

Thanks,
Chris

3

Thanks!
It turned out to be a lot more straightforward than I thought.
The key part I missed before was that the first templated bool in the TStaticDepthStencilState controls whether or not you write to the depth buffer.
I turned the interior portion of the above code into:

bool bWriteDepth = !PrimitiveSceneProxy->ShouldRenderStencilOnly();					
if (bWriteDepth)
{
    RHICmdList.SetDepthStencilState(TStaticDepthStencilState::GetRHI(), CustomDepthStencilValue);
}
else
{
    RHICmdList.SetDepthStencilState(TStaticDepthStencilState::GetRHI(), CustomDepthStencilValue);
}

(with plumbing a bool for “RenderStencilOnly” through the pipeline, of course)