Error during custom RayTracer Shader initialization

Development Rendering
1

Hi all!
I am following the tutorial to create a custom ray tracer shader, porting it to UE 5.5.1
(How to Create a Custom Ray Tracing Shader as a Plugin | Community tutorial)
I have adapted some calls to match the new API, it compiles but I am getting a runtime error
Basically in the postOpaqueRender delegate

        FRayGenTest::Execute_RenderThread(FPostOpaqueRenderParameters& Parameters)

when I add my custom pass to GraphBuilder

	// add the ray trace dispatch pass
	GraphBuilder->AddPass(
		RDG_EVENT_NAME("RayGenTest"),
		PassParameters,
		ERDGPassFlags::Compute,
		[
			PassParameters, 
			RayGenTestRGS, 
			TextureSize, 
			RTScene, 
			layerView, 
			RHIScene,
			this
		](FRHIRayTracingCommandList& RHICmdList)
		{
	...
				RHICmdList.SetRayTracingMissShader(RHIScene, RAY_TRACING_MISS_SHADER_SLOT_DEFAULT, PipeLine, 0 /* ShaderIndexInPipeline */, 0, nullptr, 0);
	
	}

that line fails, trigering an out of bounds assert
Going deeply in the engine code, this one fails

FORCEINLINE_DEBUGGABLE void SetRayTracingBindings(
	FRHIRayTracingScene* Scene, FRayTracingPipelineState* Pipeline,
	uint32 NumBindings, const FRayTracingLocalShaderBindings* InBindings,
	ERayTracingBindingType BindingType,
	bool bCopyDataToInlineStorage = true)

here

			Bindings[i].Geometry = Scene->GetInitializer().PerInstanceGeometries[Bindings[i].InstanceIndex];

cause Scene->GetInitializer().PerInstanceGeometries is empty!
Investigating a bit deeper, FRayTracingScene seems containing all the things I need, but not the FRHIRayTracingScene, so I presume I miss an initialization of this last object (but there’s nothing about it in the original tutorial so I am puzzled if I need to do it by myself or not)…any idea or suggestion?

Thanks!

2

Ok some more info here:
these init buffers are filled in

FRayTracingScene::Create

by a separate thread

// Fill instance upload buffer on separate thread since results are only needed in RHI thread
GraphBuilder.AddCommandListSetupTask(...)
{
...
FillRayTracingInstanceUploadBuffer(
				RayTracingSceneRHI,
				PreViewTranslation,
				Instances,
				InstanceGeometryIndices,
				BaseUploadBufferOffsets,
				BaseInstancePrefixSum,
				NumNativeGPUSceneInstances,
				NumNativeCPUInstances,
				MakeArrayView(InstanceUploadData, NumNativeInstances),
				MakeArrayView(TransformUploadData, NumNativeCPUInstances * 3));
...
}

so it looks like a sync issue… My plugin is marked as PostConfigInit and the shader is iexecuted during postOpaqueRender step, so I am wondering how to sync it properly

3

In the end I was able to make it work under Windows and DX12, moving from using FRHIRayTracingScene, now deprecated and probably broken, to FRHIShaderBindingTable

Basically:

FRHIShaderBindingTable* SBT = RayTracingSBT->AllocateRHI(RHICmdList, ERayTracingShaderBindingMode::RTPSO, ERayTracingHitGroupIndexingMode::Allow, 1, 0, PSOInitializer.GetMaxLocalBindingDataSize());

and use it with RHICmdLis

RHICmdList.SetRayTracingMissShader(SBT, 0, PipeLine, 0 /* ShaderIndexInPipeline */, 0, nullptr, 0);
RHICmdList.CommitShaderBindingTable(SBT);
RHICmdList.RayTraceDispatch(
			PipeLine,
			GetGlobalShaderMap(GMaxRHIFeatureLevel)->GetShader<FRayGenTestRGS>().GetRayTracingShader(),
			SBT, 
			GlobalResources, 
			TextureSize.X,
			TextureSize.Y
		);

Unfortunately using Vulkan the same code does not work, it hungs for a while and then I get VK_Device_Lost error whiel retrieving the FRDGBufferSRVRef layerView
So it’s still WIP

4

I’m having the same problem and I’m trying to use your solution, but I genuinely don’t understand what that “RayTracingSBT” is in the first line.

If it’s a FRHIShaderBindingTable object then I guess things are even more broken now because it doesn’t overload the → operator, otherwise I really don’t understand what it is.

If you could clarify it would help me alot, I’ve been stuck on trying to make that old tutorial work for the past weeks.

UPDATE: I didn’t find any method named AllocateRHI, that may be because I’m working on 5.6. I did find an AllocateTransientRHI method, so if understood correctly:

FRayTracingShaderBindingTable RayTracingSBT = FRayTracingShaderBindingTable();

FRHIShaderBindingTable* SBT = RayTracingSBT.AllocateTransientRHI(RHICmdList, ERayTracingShaderBindingMode::RTPSO, ERayTracingHitGroupIndexingMode::Allow, PSOInitializer.GetMaxLocalBindingDataSize());