Community Tutorial: How to Create a Custom Ray Tracing Shader as a Plugin

[FluffyBunnyO]

In this tutorial I will explain how to create a new ray tracing shader without touching the engine code.
We will not create our own ray tracing function, we will use the actual DirectX TraceRay() and the unreal engine API for it.
Our shader output will be a simple hit/miss shader - if our ray hit we dray white, if we miss black.

In this tutorial we will not go over what is ray tracing, accelerated structure, ray gen shader, closest hit shader etc. This knowledge is essential and should be acquired somewhere else before delving into this guide.

https://dev.epicgames.com/community/learning/tutorials/1lGL/unreal-engine-how-to-create-a-custom-ray-tracing-shader-as-a-plugin

[T.Ashby-CPP]

This is great! Thanks!

I want to mention that this doesn’t appear to be usable in 4.27. It’s 5.0 or above.

[1shenchao]

Hi，I learn your tutorial，but met a problem ‘not find shader RayTraceTestRGS’ in UE 5.3.
I have add shader source directoryMapping. Is shader function format wrong?
RAY_TRACING_ENTRY_RAYGEN(RayTraceTestRGS)
{
// reconstruct camera direction with a small (far) device z
uint2 PixelCoord = DispatchRaysIndex().xy + View.ViewRectMin.xy;
float3 WorldPosition;

}

[T.Ashby-CPP]

You need to tell the engine where to look for your Shader. I have my shaders in the plugin folder like this: “./MyPlugin/Shaders/Private/”.

Use AddShaderSourceDirectoryMapping() to tell the engine where to look for your shaders. Call this inside your StartupModule():

// Set the custom shader directory as a source directory so it can be referenced by the engine
TSharedPtr<IPlugin> plugin = IPluginManager::Get().FindPlugin(TEXT("{NAME_OF_YOUR_PLUGIN}"));

if (plugin)
{
	FString sPluginShaderDir = FPaths::Combine(plugin->GetBaseDir(), TEXT("Shaders/Private"));
	AddShaderSourceDirectoryMapping(TEXT("/Shaders"), sPluginShaderDir);
}

[T.Ashby-CPP]

Hi @FluffyBunnyO ,

I’m following your tutorial, but I’m running in UE5.3. When I got your example running, I kept getting a crash on RHICreateUnorderedAccessView. Internally, that function is running an assert with (!IsImmediate() || IsInRenderingThread() || IsInRHIThread()) and is failing the assert.

I updated the Test Runner actor to call FRayGenTest::BeginRendering() on the rendering thread instead of the update function using ENQUEUE_RENDER_COMMAND and this fixes the crash on BeginRendering().

FRayGenTestParams* pParams = new FRayGenTestParams();
pParams->pTest = &Test;
pParams->bInitialized = false;

ENQUEUE_RENDER_COMMAND(FRayGenTestParams) (
	[&pParams](FRHICommandListImmediate& RHICmdList)
	{
		pParams->bInitialized = pParams->pTest->BeginRendering();
		return true;
	});

In 5.3, GetLayerSRVChecked() was replaced with GetLayerView() which returns a different object. You have to call GetRHI() on while inside the render pass to get the TLAS from the LayerView object that was returned.

See this thread for more details on getting the TLAS: How to get the scene Accelerate Structure (TLAS) for custom Ray Tracing - #5 by TJ.Ashby-ARA

[marvinstr1 (marvinstr)]

@T.Ashby.CPP Thank you for this snippet, but I am unable to make it work (memory access violations). Do you have a complete working example for 5.3 that you can share? Thank you in advance.

[T.Ashby-CPP]

Can you copy what you’ve done so far? Maybe I can help. I’ve since made major modifications to my implementation to support my current project.

[marvinstr1 (marvinstr)]

Hi @TJ.Ashby-ARA,

Thank you for your reply, my solution is not yet on Github, but this is what I have done and it “kind off” works. However, it often crashes due to memory access violation, most likely as a result of not properly protecting the output buffer (however, I’m very new to this and have no clue how to lock it).

This is what I’ve done:

void FRayGenTest::Execute_RenderThread(FPostOpaqueRenderParameters &Parameters)
{
	if (!mCachedParams.Scene->IsCreated()) return;

	FRDGBuilder *GraphBuilder = Parameters.GraphBuilder;
	FRHICommandListImmediate &RHICmdList = GraphBuilder->RHICmdList;

	if (!(bCachedParamsAreValid && mCachedParams.RenderTarget))
	{
		return;
	}

	// Render Thread Assertion
	check(IsInRenderingThread());

	TShaderMapRef<FRayGenTestRGS> RayGenTestRGS(GetGlobalShaderMap(GMaxRHIFeatureLevel));

	bool bIsShaderValid = RayGenTestRGS.IsValid();
	if (!bIsShaderValid)
		return;

	const FIntPoint TextureSize = {mCachedParams.RenderTarget->SizeX, mCachedParams.RenderTarget->SizeY};

	FRHITextureCreateDesc TextureDesc = FRHITextureCreateDesc::Create2D(TEXT("RaytracingTestOutput"), TextureSize.X, TextureSize.Y, mCachedParams.RenderTarget->GetFormat());
	TextureDesc.AddFlags(TexCreate_ShaderResource | TexCreate_UAV | TexCreate_RenderTargetable);
	mShaderOutputTexture = RHICreateTexture(TextureDesc);

	// set shader parameters
	FRayGenTestRGS::FParameters *PassParameters = GraphBuilder->AllocParameters<FRayGenTestRGS::FParameters>();
	PassParameters->ViewUniformBuffer = Parameters.View->ViewUniformBuffer;
	PassParameters->outTex = RHICmdList.CreateUnorderedAccessView(mShaderOutputTexture);

	FTexture2DRHIRef OriginalRT = mCachedParams.RenderTarget->GetRenderTargetResource()->GetTexture2DRHI();
	FRDGTexture *CopyToRDGTexture = GraphBuilder->RegisterExternalTexture(CreateRenderTarget(OriginalRT, TEXT("RaytracingTestCopyToRT")));
	FRDGTexture *OutputRDGTexture = GraphBuilder->RegisterExternalTexture(CreateRenderTarget(mShaderOutputTexture, TEXT("RaytracingTestOutputRT")));

	FRHIRayTracingScene *RHIScene = mCachedParams.Scene->GetRHIRayTracingScene();
	FRDGBufferSRVRef layerView = mCachedParams.Scene->GetLayerView(ERayTracingSceneLayer::Base);

	GraphBuilder->AddPass(
		RDG_EVENT_NAME("RayGenTest"),
		PassParameters,
		ERDGPassFlags::Compute,
		[PassParameters, RayGenTestRGS, TextureSize, RHIScene, layerView, CopyToRDGTexture, OutputRDGTexture](FRHIRayTracingCommandList &RHICmdList)
		{
			PassParameters->TLAS = layerView->GetRHI();
			FRayTracingShaderBindingsWriter GlobalResources;
			SetShaderParameters(GlobalResources, RayGenTestRGS, *PassParameters);

			FRayTracingPipelineStateInitializer PSOInitializer;
			PSOInitializer.MaxPayloadSizeInBytes = GetRayTracingPayloadTypeMaxSize(FRayGenTestRGS::GetRayTracingPayloadType(0));
			PSOInitializer.bAllowHitGroupIndexing = false;

			// Set RayGen shader
			TArray<FRHIRayTracingShader *> RayGenShaderTable;
			RayGenShaderTable.Add(GetGlobalShaderMap(GMaxRHIFeatureLevel)->GetShader<FRayGenTestRGS>().GetRayTracingShader());
			PSOInitializer.SetRayGenShaderTable(RayGenShaderTable);

			// Set ClosestHit shader
			TArray<FRHIRayTracingShader *> RayHitShaderTable;
			RayHitShaderTable.Add(GetGlobalShaderMap(GMaxRHIFeatureLevel)->GetShader<FRayGenTestCHS>().GetRayTracingShader());
			PSOInitializer.SetHitGroupTable(RayHitShaderTable);

			// Set Miss shader
			TArray<FRHIRayTracingShader *> RayMissShaderTable;
			RayMissShaderTable.Add(GetGlobalShaderMap(GMaxRHIFeatureLevel)->GetShader<FRayGenTestMS>().GetRayTracingShader());
			PSOInitializer.SetMissShaderTable(RayMissShaderTable);

			// dispatch ray trace shader
			FRayTracingPipelineState *PipeLine = PipelineStateCache::GetAndOrCreateRayTracingPipelineState(RHICmdList, PSOInitializer);
			RHICmdList.SetRayTracingMissShader(RHIScene, 0, PipeLine, 0 /* ShaderIndexInPipeline */, 0, nullptr, 0);
			RHICmdList.RayTraceDispatch(PipeLine, RayGenTestRGS.GetRayTracingShader(), RHIScene, GlobalResources, TextureSize.X, TextureSize.Y);

			FRHICopyTextureInfo CopyInfo;
			CopyInfo.Size = FIntVector(TextureSize.X, TextureSize.Y, 0);

			RHICmdList.CopyTexture(OutputRDGTexture->GetRHI(), CopyToRDGTexture->GetRHI(), CopyInfo);
		});
}

Ultimately, though, what I am looking for is more of a replacement for the CPU based LineTrace function, because I need to cast a lot of rays (for LIDAR simulation). So I would like to in the future rewrite it a bit to use a structured buffer with the desired ray properties and input and a vector with the hit results as output.

I also found the RayTracing test case in the Unreal Engine source code, but I was not able to adapt this to my needs, because I struggled to get access to the TLAS structure.

Please let me know if you have any suggestions on how to solve the crashing issues when copying the texture buffer. Thank you for your support!

[noubernou]

This is an amazing tutorial, got it working in 5.3 with a bit of tweaking from the other comments.

Has anyone been able to get custom intersection shader working in Unreal though? I am at a loss, it’s definitely compiling the shader, and I have added it to the global shader definition in the hit group, but it seems to do nothing. I can make it empty or always report a hit and nothing seems to happen.

Any hints would be greatly appreciated!

[T.Ashby-CPP]

I would recommend doing the texture copy outside of this render pass.

Here is what my code looks like

// Add the ray trace dispatch pass
Parameters.GraphBuilder->AddPass(
	FRDGEventName(TEXT("RTS")),
	PassParameters,
	ERDGPassFlags::Compute,
	[PassParameters, RayGenTestRGS, pShaderMap, RHIScene, this](FRHIRayTracingCommandList& RHICmdList)
	{
		FRDGBufferSRVRef layerView = Scene->GetLayerView(ERayTracingSceneLayer::Base);
		if (layerView)
		{
			PassParameters->TLAS = layerView->GetRHI();

			// Shader Parameters
			FRayTracingShaderBindingsWriter GlobalResources;
			SetShaderParameters(GlobalResources, RayGenTestRGS, *PassParameters);

			// Init Shaders
			FRayTracingPipelineStateInitializer PSOInitializer;
			PSOInitializer.MaxPayloadSizeInBytes = GetRayTracingPayloadTypeMaxSize(FCustomRGS::GetRayTracingPayloadType(0));
			PSOInitializer.bAllowHitGroupIndexing = false;

			// Set RayGen shader
			TArray<FRHIRayTracingShader*> RayGenShaderTable;
			RayGenShaderTable.Add(RayGenTestRGS.GetRayTracingShader());
			PSOInitializer.SetRayGenShaderTable(RayGenShaderTable);

			// Set ClosestHit shader
			TArray<FRHIRayTracingShader*> RayHitShaderTable;
			RayHitShaderTable.Add(pShaderMap->GetShader<FCustomCHS>().GetRayTracingShader());
			PSOInitializer.SetHitGroupTable(RayHitShaderTable);

			// Set Miss shader
			TArray<FRHIRayTracingShader*> RayMissShaderTable;
			RayMissShaderTable.Add(pShaderMap->GetShader<FCustomMS>().GetRayTracingShader());
			PSOInitializer.SetMissShaderTable(RayMissShaderTable);

			// Dispatch ray trace shader
			FRayTracingPipelineState* PipeLine = PipelineStateCache::GetAndOrCreateRayTracingPipelineState(RHICmdList, PSOInitializer);
			RHICmdList.SetRayTracingMissShader(RHIScene, 0, PipeLine, 0, 0, nullptr, 0);
			RHICmdList.RayTraceDispatch(PipeLine, RayGenTestRGS.GetRayTracingShader(), RHIScene, GlobalResources, TargetSize.X, TargetSize.Y);
		}
	}
);

CopyOutputToRT(Parameters.GraphBuilder, RenderTarget);

void FCustomRTS::CopyOutputToRT(FRDGBuilder* inGraphBuilder)
{
	FTexture2DRHIRef OriginalRT = RenderTarget->GetRenderTargetResource()->GetTexture2DRHI();
	FRDGTexture* OutputRDGTexture = inGraphBuilder->RegisterExternalTexture(CreateRenderTarget(ShaderOutputTexture, TEXT("RTSOutputRT")));
	FRDGTexture* CopyToRDGTexture = inGraphBuilder->RegisterExternalTexture(CreateRenderTarget(OriginalRT, TEXT("RTS_ToRT")));
	FRHICopyTextureInfo CopyInfo;
	CopyInfo.Size = FIntVector(RenderTarget->SizeX, RenderTarget->SizeY, 0);
	AddCopyTexturePass(*inGraphBuilder, OutputRDGTexture, CopyToRDGTexture, CopyInfo);
}

[T.Ashby-CPP]

@noubernou Does your shader implementation macro for the hit shader look like this?

IMPLEMENT_GLOBAL_SHADER(FCustomCHS, "/Shaders/CustomRTS.usf", "closestHit=CustomCHS", SF_RayHitGroup);

And does the name of the function in your shader code match what you have in the IMPLEMENT_GLOBAL_SHADER() macro? (Mine is “CustomCHS”)

[shader("closesthit")]
void CustomCHS(inout FMinimalPayload data, BuiltInTriangleIntersectionAttributes attribs) // Called when the ray trace hit something
{
	data.HitT = RayTCurrent();
}

Also make sure the main ray tracing shader has the SF_RayGen flag set.

IMPLEMENT_GLOBAL_SHADER(FCustomRGS, "/Shaders/CustomRTS.usf", "CustomRGS", SF_RayGen);

[noubernou]

Hi @T.Ashby-CPP, thanks for responding!

I might have not been super clear in my post, my ray gen and closest hit shaders are working fine, I can interpret results from closest hit, any hit, and miss shaders and my generation shader is also working (obviously).

I am looking to add a custom intersection shader that works on the initial AABB hits, similar to the DirectX12 Procedural Geometry sample here: DirectX-Graphics-Samples/Samples/Desktop/D3D12Raytracing/src/D3D12RaytracingProceduralGeometry/readme.md at master · microsoft/DirectX-Graphics-Samples · GitHub

My hit group definition looks like this:

IMPLEMENT_GLOBAL_SHADER(FRayGenTestCHS, MY_SHADER_PATH, "closestHit=RayTraceTestCHS anyhit=RayTraceTestAHS intersection=RayTraceTestIS", SF_RayHitGroup);

And my my shaders look like this:

[shader("miss")]
void RayTraceTestMS(inout FPackedMaterialClosestHitPayload data)
{
	data.SetMiss();
}

[shader("closesthit")]
void RayTraceTestCHS(inout FPackedMaterialClosestHitPayload data, FRayTracingIntersectionAttributes attribs)
{
	data.HitT = attribs.GetBarycentrics().x;
}

[shader("anyhit")]
void RayTraceTestAHS(inout FPackedMaterialClosestHitPayload data, FRayTracingIntersectionAttributes attribs)
{
	IgnoreHit(); // need all the hits along the ray
}

[shader("intersection")]
void RayTraceTestIS()
{
	float THit = RayTCurrent();
	FRayTracingIntersectionAttributes t = (FRayTracingIntersectionAttributes)0;
	t.SetBarycentrics(float2(1.0f, 0.0f));
	ReportHit(THit, 0, t);
}

You can see in the intersection shader I am trying to set the Barycentrics X coordinate to 1.0 which in my shader I use to color the red channel in my output texture, so things should just be solid red if I hit an AABB, but when I run this it looks like the default built in intersection shader continues to run and each face is shaded with the actual X value of the barycentric coordinates.

I’ve dug through the Unreal source code and they implement custom intersection shaders for hair materials and sparse voxels, and I have tried to implement my custom intersection shader setup in the same way but alas it never seems to run.

[T.Ashby-CPP]

I tried it myself and ran into the same issue. I didn’t see anything special/different about the examples in the engine. Sorry, I’m not able to help with this one.

I would be curious to know the fix to this as well if someone else can find it!

[T.Ashby-CPP]

If it’s crashing on startup in PIE, then you need to delay the initialization a little. In the guide, it delays for one second, but you may need to delay a little longer depending on your system.

It doesn’t crash in stand-alone, but you still need to delay the initialization a bit on a stand-alone. For me, I just delayed 0.1 seconds and that was enough.

[stellarhomes]

Both Nvidia and a single modder have created ray tracing shaders to improve the game’s lighting dramatically.

[marvinstr1 (marvinstr)]

@T.Ashby-CPP Yeah, I did sort of manage to make it work… Introducing the delay seems to be mandatory, even when checking if the raytracing scene has been populated and is valid.

That said, it still seems to be still quite crash prone, basically when trying to view the RT or the actor details in the editor, it will crash due to trying to access the same output texture.

When I get to it, I will try to instead of drawing to a texture, simply putting the output of the rays into a TArray, this will maybe allow me to lock the array or figure out some other way to protect the data from being accessed simultaneously.

[Caleb_schae]

I was curious if any of you knew of existing vulkan API’s that exist within Unreal Engine to do a ray trace. I know this example uses DirectX, but I am trying to write a shader for Linux in UE 5.3. The current version allows for hardware ray tracing but I am unaware if there are API’s to call a ray without having to do something external to make it work. If anyone has any suggestions, I would definitely appreciate it!