I’m trying to create my own custom raytracing surface shader in UE 5.7.4.
Is there any supported path in UE 5.7 to supply a custom ray tracing hit group shader for a specific primitive (or general static mesh) without modifying engine source? I essentially want to be able have complete control over the reflected ray paths.
The two main points I’ve tried to hook into are GetDynamicRayTracingInstances and FRayTracingLocalShaderBindings. I’ll explain briefly what I’ve been trying to do so far:
GetDynamicRayTracingInstances
This is a virtual function on FPrimitiveSceneProxy that gets called each frame to gather ray tracing instances for the TLAS. I subclassed FStaticMeshSceneProxy and overrode this function hoping to swap in our custom hit shader for a mesh. The problem is that the instances it produces use FRayTracingInstance, which takes FMeshBatch arrays for its material slots — and the hit shader is determined downstream by the renderer when it processes those mesh batches through the material’s compiled shader map. There is no field on FRayTracingInstance or FMeshBatch that lets you say “use this specific global shader as the hit group” — it always goes through the material system.
FRayTracingLocalShaderBindings
This struct exists in the RHI layer — it represents a custom shader binding for a specific geometry segment in the SBT. I can see it being used internally by the renderer when building the shader binding table. However there is no exposed path from GetDynamicRayTracingInstances or anywhere else accessible from proxy/plugin code to inject an FRayTracingLocalShaderBindings that overrides the material’s hit shader for a specific primitive. The connection between the instance collector and the SBT construction is entirely internal to the renderer.
Anyone got any insights into how to properly go about this or whether this is just inaccessible in UE 5.7.4 without modifying the engine source (which I’d definitely rather avoid! Haha)