UE 5.6.1: Crash when a Deferred Decal (Translucent) base material has a Nanite Override (Masked Surface) and the mesh has Nanite + Ray Tracing enabled.

Development Asset Creation
, ,
1

Looks like a conflict between base material = Deferred Decal / Translucent and Nanite Override = Masked Surface when Nanite + Ray Tracing are enabled. In our main project (migrated 5.4 → 5.6) crashes began on map open. In a clean 5.6.1 project the minimal setup above reproduces the crash as well.

Two different call stacks observed in the real project (5.6/5.6.1):

  • Path A (during caching):
  • FPrimitiveSceneInfo::CacheRayTracingPrimitives → CacheRayTracingPrimitive → CacheRayTracingMeshBatch → check(!bMustEmitCommand || CommandContext.CommandIndex >= 0).
  • Path B (during gather):
  • RayTracing::GatherRelevantStaticPrimitives → FPrimitiveSceneInfo::GetRayTracingLODData(0) → TArray::operator[] → ...
  • (array OOB when RayTracingLODData is empty, i.e. Nanite RT commands were never generated).

In the clean repro, the crash also happens as soon as the mesh with the above material setup is viewed/loaded with RT+Nanite enabled.

We suspect the mesh gets classified by the base Deferred Decal domain, while the Nanite path tries to use the Override Surface material, leading to inconsistent flags and missing Nanite RT mesh commands, then an unchecked access.

2

Steps to Reproduce
Steps to Reproduce

  1. Create a new UE 5.6.1 blank project (DX12).
  2. Enable ray tracing. Either:
  • Project Settings → Rendering → Ray Tracing (Enable), Support Hardware Ray Tracing (Enable), or
  • DefaultEngine.ini:

[/Script/Engine.RendererSettings] r.RayTracing=True1. Make two materials:

  • Base: Domain = Deferred Decal, Blend Mode = Translucent (any simple setup is fine). Used with Nanite - True.
  • Nanite: Domain = Surface, Blend Mode = Masked (default lit, trivial graph). Used with Nanite - True.
  1. Make material instances: Base_Inst (from Base) and Nanite_Inst (from Nanite).
  2. In Base_Inst, set Nanite Override Material = Nanite_Inst.
  3. Create a simple Static Mesh (e.g. default Cube). Enable Nanite on the mesh and set the mesh/component to be Visible in Ray Tracing.
  4. Assign Base_Inst to the mesh/component.
  5. Open the level (or open the mesh asset in the editor) with RT enabled.
3

Hi! Possibly related - we did recently fix two crashes in this area with similar callstacks in these changelists:

CL#43313984 Update PrimitiveRayTracingData when DrawnInGame flag is modified.

CL#43397281 [HWRT] Update the entire ray tracing cached state when updating a primitive.

Will test your repro case and see if it is in a similar area.

4

I’ve verified the issue still occurs in latest, so the two CLs I mentioned don’t appear to fix it. I’ve created the following internal issue for tracking which should be visible publicly soon Unreal Engine Issues and Bug Tracker (UE\-313854)