Custom Ray Tracing Hit Shader Per-Primitive in UE 5.7 — Where to Inject?

Development Rendering
,
1

Background

I’m trying to bind a custom ray tracing closest-hit shader to a specific mesh primitive, so that rays hitting that mesh execute my CHS (which traces a secondary ray in a custom direction) instead of the material’s compiled hit shader. I’m not modifying engine source.

What I’ve achieved so far

I have a working FGlobalShader subclass declared as SF_RayHitGroup that compiles successfully from a game project module (with PostConfigInit loading phase). I have a FStaticMeshSceneProxy subclass that forces the primitive onto the dynamic ray tracing path via:

cpp

ERayTracingPrimitiveFlags FCustomRayTracingProxy::GetCachedRayTracingInstance(
    FRayTracingInstance& OutRayTracingInstance)
{
    return ERayTracingPrimitiveFlags::Dynamic;
}

I have a working FRayTracingMeshCommandContext subclass that correctly intercepts FRayTracingMeshCommand and swaps MaterialShader to my custom CHS:

cpp

FRayTracingMeshCommand& FCustomRayTracingMeshCommandContext::AddCommand(
    const FRayTracingMeshCommand& Initializer)
{
    const int32 index = _command_storage.AddElement(Initializer);
    FRayTracingMeshCommand& new_command = _command_storage[index];
    new_command.GeometrySegmentIndex = _segment_index;
    new_command.MaterialShader = _custom_shader; // <-- injection
    return new_command;
}

The problem — timing of FRayTracingSBTAllocation

To construct FDynamicRayTracingMeshCommandContext (or my subclass of FRayTracingMeshCommandContext), I need an FRayTracingSBTAllocation* which comes from:

cpp

FRayTracingSBTAllocation* sbt_allocation = RayTracingSBT.AllocateDynamicRange(
    ERayTracingShaderBindingLayerMask::Base, NumSegments);

However AllocateDynamicRange requires bStaticAllocationsLocked == true, which is set by RayTracingScene.LockCachedInstances() in FinishGatherInstances — which runs after GetDynamicRayTracingInstances is called. Calling AllocateDynamicRange from within GetDynamicRayTracingInstances triggers:

Ensure condition failed: bStaticAllocationsLocked
[RayTracingShaderBindingTable.cpp Line 531]

What I can see in the engine

In RayTracing.cpp, DispatchRayTracingMeshBatchTask constructs FDynamicRayTracingMeshCommandContext with a valid SBTAllocation from FRayTracingMeshBatchWorkItem::SBTAllocation, which was set during AddInstancesToScene — after locking. The mesh batch processing happens on a task thread at that point, well after GetDynamicRayTracingInstances.

The question

Is there a supported hook point where I can:

  1. Supply a custom FRayTracingMeshCommandContext (with my shader swap) for processing a specific primitive’s mesh batches

  2. At a point in the pipeline where a valid FRayTracingSBTAllocation is available

Or alternatively — is there a way to supply a pre-built FRayTracingMeshCommand with a custom MaterialShader directly to FRayTracingShaderBindingDataOneFrameArray on FViewInfo, bypassing FRayTracingMeshProcessor entirely?

The full reproduction is a minimal project-code implementation — happy to share the source if helpful.