Error during custom RayTracer Shader initialization

Development Rendering
1

Hi all!
I am following the tutorial to create a custom ray tracer shader, porting it to UE 5.5.1
(How to Create a Custom Ray Tracing Shader as a Plugin | Community tutorial)
I have adapted some calls to match the new API, it compiles but I am getting a runtime error
Basically in the postOpaqueRender delegate

        FRayGenTest::Execute_RenderThread(FPostOpaqueRenderParameters& Parameters)

when I add my custom pass to GraphBuilder

	// add the ray trace dispatch pass
	GraphBuilder->AddPass(
		RDG_EVENT_NAME("RayGenTest"),
		PassParameters,
		ERDGPassFlags::Compute,
		[
			PassParameters, 
			RayGenTestRGS, 
			TextureSize, 
			RTScene, 
			layerView, 
			RHIScene,
			this
		](FRHIRayTracingCommandList& RHICmdList)
		{
	...
				RHICmdList.SetRayTracingMissShader(RHIScene, RAY_TRACING_MISS_SHADER_SLOT_DEFAULT, PipeLine, 0 /* ShaderIndexInPipeline */, 0, nullptr, 0);
	
	}

that line fails, trigering an out of bounds assert
Going deeply in the engine code, this one fails

FORCEINLINE_DEBUGGABLE void SetRayTracingBindings(
	FRHIRayTracingScene* Scene, FRayTracingPipelineState* Pipeline,
	uint32 NumBindings, const FRayTracingLocalShaderBindings* InBindings,
	ERayTracingBindingType BindingType,
	bool bCopyDataToInlineStorage = true)

here

			Bindings[i].Geometry = Scene->GetInitializer().PerInstanceGeometries[Bindings[i].InstanceIndex];

cause Scene->GetInitializer().PerInstanceGeometries is empty!
Investigating a bit deeper, FRayTracingScene seems containing all the things I need, but not the FRHIRayTracingScene, so I presume I miss an initialization of this last object (but there’s nothing about it in the original tutorial so I am puzzled if I need to do it by myself or not)…any idea or suggestion?

Thanks!

2

Ok some more info here:
these init buffers are filled in

FRayTracingScene::Create

by a separate thread

// Fill instance upload buffer on separate thread since results are only needed in RHI thread
GraphBuilder.AddCommandListSetupTask(...)
{
...
FillRayTracingInstanceUploadBuffer(
				RayTracingSceneRHI,
				PreViewTranslation,
				Instances,
				InstanceGeometryIndices,
				BaseUploadBufferOffsets,
				BaseInstancePrefixSum,
				NumNativeGPUSceneInstances,
				NumNativeCPUInstances,
				MakeArrayView(InstanceUploadData, NumNativeInstances),
				MakeArrayView(TransformUploadData, NumNativeCPUInstances * 3));
...
}

so it looks like a sync issue… My plugin is marked as PostConfigInit and the shader is iexecuted during postOpaqueRender step, so I am wondering how to sync it properly

3

In the end I was able to make it work under Windows and DX12, moving from using FRHIRayTracingScene, now deprecated and probably broken, to FRHIShaderBindingTable

Basically:

FRHIShaderBindingTable* SBT = RayTracingSBT->AllocateRHI(RHICmdList, ERayTracingShaderBindingMode::RTPSO, ERayTracingHitGroupIndexingMode::Allow, 1, 0, PSOInitializer.GetMaxLocalBindingDataSize());

and use it with RHICmdLis

RHICmdList.SetRayTracingMissShader(SBT, 0, PipeLine, 0 /* ShaderIndexInPipeline */, 0, nullptr, 0);
RHICmdList.CommitShaderBindingTable(SBT);
RHICmdList.RayTraceDispatch(
			PipeLine,
			GetGlobalShaderMap(GMaxRHIFeatureLevel)->GetShader<FRayGenTestRGS>().GetRayTracingShader(),
			SBT, 
			GlobalResources, 
			TextureSize.X,
			TextureSize.Y
		);

Unfortunately using Vulkan the same code does not work, it hungs for a while and then I get VK_Device_Lost error whiel retrieving the FRDGBufferSRVRef layerView
So it’s still WIP