Extending Mesh Decals to work with Custom Stencil

Development Asset Creation
, ,
1

Hi,

We are attempting to use Mesh Decals but are running into a few limitations which are problematic for us. Looking at the engine code, they seem avoidable, so we wanted to ask about the rationale for how things are structured today.

We are attempting to replicate the standard decal projection using the WorldPositionBehindTranslucency and then mapping the WorldPos -> Instance Local Space to compute the UV. This works, however, it’s not possible to use material expressions like UMaterialExpressionObjectLocalBounds/UMaterialExpressionBounds/UMaterialExpressionPreSkinnedLocalBounds in decals, so we need to specify the mesh bounds manually. After removing the explicit blocking in MaterialExpressions.cpp and FHLSLMaterialTranslator::CheckPrimitivePropertyCompatibity ffor MD_DeferredDecal everything seems to work fine, so we were wondering what the purpose of these checks are?

Secondly our game has a UI effect based on Custom Stencil values. The bOutputTranslucentVelocity=true + OpacityMaskClipValue works here and the stencil write can be masked to a region of the decal, however practically that region depends on the output of WorldPositionBehindTranslucency which depends on SceneDepth. CustomDepth renders after the depth pre-pass so this data should be available, however it uses the generic FDepthOnlyPS shader which sets SCENE_TEXTURES_DISABLED=1 (since generally they are not, for example rendering shadow maps). Locally we were able to fix this by adding a variant of FDepthOnlyPS specific to FCustomDepthPassMeshProcessor which sets SCENE_TEXTURES_DISABLED=0 if ShouldForceFullDepthPass(Parameters.Platform), which for our project is true (and I suspect many projects also). Is this something Epic would consider supporting or are there other side-effects doing this we should be aware of? Maybe we need a more granular version of SCENE_TEXTURES_DISABLED which only allows SceneDepth to be safe?

Thanks,

Lucas

[Attachment Removed]

2

Steps to Reproduce

  • Create a decal material which uses the built-in WorldPositionBehindTranslucency Material Function to project itself onto the underlying surface and outputs an Opacity value where the decal is present using Opacity Mask Clip value.
  • Assign it on a UStaticMesh to create a mesh decal.
  • Enable custom depth/stencil writes on the UStaticMeshComponent
    [Attachment Removed]
3

Hi Lucus,

For the mesh decal question the short answer is that you should be able to remove the blocking with some caveats.

The material Deferred Decal domain generally applies to both mesh decals and UDecalComponents. For historical reasons the decal components aren’t included in the GPU scene primitives. So these don’t have straight forward access to the standard primitive data such as LocalObjectBoundsMin/Max, InstanceLocalBoundsCenter/Extent, and PreSkinnedLocalBoundsMin/Max.

I think that the decal component shaders at some point would not compile if they were allowed to use the UMaterialExpressionObjectLocalBounds/UMaterialExpressionBounds/UMaterialExpressionPreSkinnedLocalBounds expressions, but now they would just return zero defaults. So if you are happy with any decal components silently behaving in that way, then you can strip out the MD_DeferredDecal checks in the translator.

For the custom depth pass question, what you are doing makes some sense. I think that there is a similar issue with the single layer water depth pass where we need to set SCENE_TEXTURES_DISABLED=0 too.

One thought that comes to mind is that you might need to take care that your FDepthOnlyPS variant is well separated, so that shadow passes don’t get affected.

Also I agree that your suggestion for a more granular SCENE_TEXTURES_DISABLED starts to make sense. Not only so that it could be on in more scenarios, but also to make it clear that things other than depth aren’t available depending on where you are in the frame. I think for the CustomDepth pass, only Depth would be available?

Best regards,

Jeremy

[Attachment Removed]

4

Hi Jeremy,

Thanks for taking a look!

I think that the decal component shaders at some point would not compile if they were allowed to use the UMaterialExpressionObjectLocalBounds/UMaterialExpressionBounds/UMaterialExpressionPreSkinnedLocalBounds expressions, but now they would just return zero defaults. So if you are happy with any decal components silently behaving in that way, then you can strip out the MD_DeferredDecal checks in the translator.

Thanks for the background here. Good to know about the GPU Scene limitations but yeah, that isn’t an issue for us since we already need to have a static switch in the material to do a UDecalComponent-like projection using the mesh decal mesh (which practically is just a cube) so those nodes are only used in that path.

Really, we just wanted a UDecalComponent that can write to custom depth, but as you mention there is some legacy stuff related to those. It would be great if mesh decals and normal decals were unified someday since you can essentially recreate a UDecalComponent with a mesh decal today, you just need to manually do the projection in the material. Having a UDecalComponent be a UPrimitiveComponent which made a mesh decal under the hood and possibly having some flag on the material to opt into DeferredDecal.usf-style projection automatically for a mesh decal would be ideal and maybe allow some renderer paths to be removed.

One thought that comes to mind is that you might need to take care that your FDepthOnlyPS variant is well separated, so that shadow passes don’t get affected.

I was also nervous about this, so we made a derived ShaderType from FDepthOnlyPS, FCustomDepthOnlyPS which is only used by FCustomDepthPassMeshProcessor and not the others. There are probably lots of ways to do this (FPermutationDomain, etc.) but I was going for the minimal engine change. :slight_smile:

Also I agree that your suggestion for a more granular SCENE_TEXTURES_DISABLED starts to make sense. Not only so that it could be on in more scenarios, but also to make it clear that things other than depth aren’t available depending on where you are in the frame. I think for the CustomDepth pass, only Depth would be available?

Possibly if the shader needs to change behavior, but debugging this more, it seems like the ESceneTextureSetupMode flags in SetupSceneTextureUniformParameters takes care of it so everything else other than depth gets a dummy texture and it kind of just does the right thing already, which is nice. You don’t end up with invalid RDG dependencies or anything like that even if you try and read other scene textures in the material. I guess that also makes me wonder what the purpose of SCENE_TEXTURES_DISABLED is if the code is already tracking which ones are available and only binding valid textures for a given pass. Maybe it predates this?

Thanks,

Lucas

[Attachment Removed]