[Help needed]custom RayTrace feature

I am trying to use TraceRay method to do some custom work.

When I need the world normal of the hit point, I found an example code in PathTracingMainRG, which uses FPackedMaterialClosestHitPayload to do the trace.

But in my case, my FPackedMaterialClosestHitPayload gets nothing but only a HitT value. The other member variables of FPackedMaterialClosestHitPayload are all zero.

I wonder how the method TraceRay works, but I get this when I search with the keyword:

/**
***********************************************************************************************************************
* @brief
*    Convenience macro for calling TraceRays that uses the hit token
*
***********************************************************************************************************************
*/
#define AmdTraceRay(accelStruct,                    \
                    rayFlags,                       \
                    instanceInclusionMask,          \
                    rayContributionToHitGroupIndex, \
                    geometryMultiplier,             \
                    missShaderIndex,                \
                    ray,                            \
                    payload,                        \
                    token)                          \
AmdSetHitToken(token);                              \
TraceRay(accelStruct,                               \
         rayFlags,                                  \
         instanceInclusionMask,                     \
         rayContributionToHitGroupIndex,            \
         geometryMultiplier,                        \
         missShaderIndex,                           \
         ray,                                       \
         payload);                                  \


#endif // _AMDEXTD3DSHADERINTRINICS_HLSL

I cannot know how it processes the different Payload structs since they are in a relationship of inheritance.

I think the key may be the resources bound to the shader. Path tracer may need some resources to get the extra infomation. Now the situation is, only FPackedMaterialClosestHitPayload has the normal info, and only path tracer uses this Payload.

There are two ways now. One is figuring out the path tracer. The other is trying getting WorldNormal from the normal Payload such as FDefaultPayload. Or is it possible to get the TraceRay API more clear?

I am actually struggling with the same, I did a bit more progress, instead of using the FDefaultPayload, I managed to create my own, the procedure is the same, you need a child of FGlobalShader as usual, for the implementation you need to use the SF_RayHitGroup,

IMPLEMENT_GLOBAL_SHADER(FRainbowCurvatureHitCHS, “/Rainbow/ImageProcessing/RainbowCurvature.usf”, “closesthit=CurvatureHitCHS anyhit=CurvatureHitAHS”, SF_RayHitGroup);

It seems that both the closesthit and the anyhit can be implemented with the same shader.

Now before calling the RHICmdList.RayTraceDispatch you should set your hit shaders into the FRayTracingPipelineStateInitializer, here I have my example,

I even manage to replace the miss shader, which is the same procedure as you can see.

My only struggle now is inside the closetHit shader, I am trying to get back the normal and position from the hit, but at least I can tell you that the TraceRay function is giving me back right Payload info, my variables are correctly populated on the closest hit shader.

I am close I know.

Also sorry for the late answer, but you know Epic never helps with very technical stuff, so I answer every time I can

EDIT: I manage to get the world position like this:

Finally solved, it is not possible to create a custom hit shader at this moment, due the fact that ByteAddressBuffer for the Vertex buffer cannot be bind without tricks, so reading the GPU lightmass plugin I found out you need to create a child of FRayTracingMeshProcessor, this guy is in charge to generate the a mesh command when calling the function BuildRayTracingMeshCommands, from this mesh command you can hack your way in and create the necessary binds:

All that just to bind the vertex buffer, unfortunately creating the MeshProcessor will force you to create a mesh shader just to work, you don’t really use the hit shader declared for it, it is just a dummy to make your mesh processor work, you can bind an existing UE’s like this:

That will finally give you access to the hit shader from UE and the payload will be finally populated like this:

Very loooong run to fix such simple thing, on the GPU lightmass code there is a TODO note where the author say they will maybe make the MeshCommand a virtual function so you don’t need the MeshProcessor

2 Likes

Hi!
Thanks for sharing!
I was wondering if you could somehow share the whole code for initializing the shader?
There is a sever lack of information on how to do so.
I’ve collected all kinds of information pieces about initializing a shader in UE5, but non of which are for a rayGen/miss/anyhit etc.
The examples in the engine are sometimes too complex and requires private access that my plugin will not have.

1 Like

Hey FluffyBunnyO

I know, it is kinda difficult learning, as weird as it sounds this time, UE has a marvelous example as a unit test (I know, a full example, what a shock!!!) check for RayTracingTestBed.cpp, it is the simplest example about raytracing you will ever find.

Thanks @ZkarmaKun for the response and sorry for the late one of mine.
I’ve looked into it, that is shocking. However, it handles everything except my issue: where do I get the accelerated structure (AKA TLAS) from.
In the example, the unit test creates it’s own unique geometry to test all the RT callbacks (miss, anyhit, closest hit etc.). From this geometry it generates the TLAS.
In other implementation in UE, the TLAS is injected into the render function as an input parameter. When I try to trace back the origin of the input TLAS, it only leads me to private interface which I do not have access to from my plugin.

if you want to access the TLAS from unreal’s scene I afraid your options will be limited, the only way AFAIK is by creating a FSceneViewExtensionBase class, this will allow you to override some passes in the post process chain, giving you access to the FSceneView and finally to the TLAS from there, but like I said, it is limited since it does not allow you do override all states like the post process materials chain. You can take a look at ColorCorrectRegionsSceneViewExtension.h to know how is implemented.

In my case I am creating my own TLAS like in the previous example! Also if you want to check a more complete example about this, you can take a look at the GPULightmass which is a bit huge, but it also does the same, it creates its own TLAS for the texture baking

@ZkarmaKun Ok so I think I got it. I didn’t write anything down yet, but I have an idea on what to do.
I need to create a top level TLAS using the scene geometry instances. These instances can be access via the scene → render scene → RayTracingScene → Instances (UE 5.1).

Thank you! I would have spent days looking for the scene TLAS instead of creating my own without your answer.
Hopefully I’ll get this one done and post a small guide on how to implement a RT shader.

@ZkarmaKun ok so I got it up and running, but I get a crash deep inside the d12 API.

This my shader:

#include "/Engine/Public/Platform.ush"

#include "/Engine/Private/Common.ush"
#include "/Engine/Private/RayTracing/RayTracingCommon.ush"
#include "/Engine/Private/RayTracing/RayTracingHitGroupCommon.ush"


RWTexture2D<float4> outTex;
float3 wsCamPos;
float3 wsCamU, wsCamV, wsCamW; // camera up, right, and forward vectors
RaytracingAccelerationStructure TLAS;

struct Payload
{
    float3 color;
};

[shader("raygeneration")]
void RayTraceTestRGS()
{
    uint2 curPixel = DispatchRaysIndex().xy;
    uint2 totalPixels = DispatchRaysDimensions().xy;
    float2 pixelCenter = (curPixel + float2(0.5, 0.5)) / totalPixels;
    float2 ndc = float2(2, -2) * pixelCenter + float2(-1, 1); // Normalized device coords
    float3 pixelRayDir = ndc.x * wsCamU + ndc.y * wsCamV + wsCamW;
    
    RayDesc ray;
    ray.Origin = wsCamPos;
    ray.Direction = normalize(pixelRayDir);
    ray.TMin = 0.0f;
    ray.TMax = 1e+38f;
    
    Payload payload = (Payload) 0;
    TraceRay(TLAS, 
            0, 
            0xFF,
            0,
            2,
		    0, 
		    ray, 
		    payload);
         
    //outTex[curPixel] = payload.IsHit() ? float4(1, 1, 1, 1) : float4(0, 0, 0, 1);
    outTex[curPixel] = float4(payload.color, 1);
}

[shader("miss")]
void RayTraceTestMS(inout Payload data)
{
    //data.SetMiss();
    data.color = 0;
}

[shader("closesthit")]
void RayTraceTestCHS(inout Payload data, BuiltInTriangleIntersectionAttributes attribs)
{
    //data.HitT = 1.0f;
    data.color = 0;
}

This is my renderer callback

void RayGenTest::Execute_RenderThread(FRDGBuilder&  builder, const FSceneTextures& SceneTextures)
#if RHI_RAYTRACING
{
	FRHICommandListImmediate& RHICmdList = builder.RHICmdList;
	//If there's no cached parameters to use, skip
	//If no Render Target is supplied in the cachedParams, skip
	if (!(bCachedParamsAreValid && cachedParams.RenderTarget))
	{
		return;
	}

	//Render Thread Assertion
	check(IsInRenderingThread());

	//If the render target is not valid, get an element from the render target pool by supplying a Descriptor
	if (!ShaderOutput.IsValid())
	{
		//UE_LOG(LogTemp, Warning, TEXT("Compute Shader Output Not Valid; re-generating"));
		FIntPoint Size = cachedParams.GetRenderTargetSize();
		FPooledRenderTargetDesc ComputeShaderOutputDesc(FPooledRenderTargetDesc::Create2DDesc(cachedParams.GetRenderTargetSize(), cachedParams.RenderTarget->GetRenderTargetResource()->TextureRHI->GetFormat(), FClearValueBinding::None, TexCreate_None, TexCreate_ShaderResource | TexCreate_UAV, false));
		ComputeShaderOutputDesc.DebugName = TEXT("ForceFieldCS_Output_RenderTarget1");
		GRenderTargetPool.FindFreeElement(RHICmdList, ComputeShaderOutputDesc, ShaderOutput, TEXT("ForceFieldCS_Output_RenderTarget2"));
	}

	FRHIRayTracingScene* RHIScene = cachedParams.Scene->GetRHIRayTracingScene();

	FRayGenTestRGS::FParameters *PassParameters = builder.AllocParameters<FRayGenTestRGS::FParameters>();
	PRAGMA_DISABLE_DEPRECATION_WARNINGS // Missing API for UAV prevents me from using the UAV.
	PassParameters->outTex = ShaderOutput->GetRenderTargetItem().UAV;
	PRAGMA_ENABLE_DEPRECATION_WARNINGS
	PassParameters->wsCamPos = static_cast<FVector3f>(cachedParams.CameraTransform.GetTranslation());
	PassParameters->wsCamU = static_cast<FVector3f>(cachedParams.CameraTransform.GetRotation().GetUpVector());
	PassParameters->wsCamV = static_cast<FVector3f>(cachedParams.CameraTransform.GetRotation().GetRightVector());
	PassParameters->wsCamW = static_cast<FVector3f>(cachedParams.CameraTransform.GetRotation().GetForwardVector());
	PassParameters->TLAS = cachedParams.Scene->GetLayerSRVChecked(ERayTracingSceneLayer::Base);

	TShaderMapRef<FRayGenTestRGS> RayGenTestRGS(GetGlobalShaderMap(GMaxRHIFeatureLevel));

	FIntPoint TextureSize = {cachedParams.RenderTarget->SizeX, cachedParams.RenderTarget->SizeY};
	builder.AddPass(
		RDG_EVENT_NAME("RayGenTest"),
		PassParameters,
		ERDGPassFlags::Compute,
		[PassParameters, RayGenTestRGS, TextureSize, RHIScene](FRHIRayTracingCommandList& RHICmdList)
		{
			FRayTracingShaderBindingsWriter GlobalResources;
			SetShaderParameters(GlobalResources, RayGenTestRGS, *PassParameters);

			FRayTracingPipelineStateInitializer PSOInitializer;
			PSOInitializer.MaxPayloadSizeInBytes = RAY_TRACING_MAX_ALLOWED_PAYLOAD_SIZE;
			PSOInitializer.bAllowHitGroupIndexing = false;

			// Set RayGen shader
			TArray<FRHIRayTracingShader*> RayGenShaderTable;
			RayGenShaderTable.Add(GetGlobalShaderMap(GMaxRHIFeatureLevel)->GetShader<FRayGenTestRGS>().GetRayTracingShader());
			PSOInitializer.SetRayGenShaderTable(RayGenShaderTable);

			// Set ClosestHit shader
			TArray<FRHIRayTracingShader*> RayHitShaderTable;
			RayHitShaderTable.Add(GetGlobalShaderMap(GMaxRHIFeatureLevel)->GetShader<FRayGenTestCHS>().GetRayTracingShader());
			PSOInitializer.SetHitGroupTable(RayHitShaderTable);
			
			// Set Miss shader
			TArray<FRHIRayTracingShader*> RayMissShaderTable;
			RayMissShaderTable.Add(GetGlobalShaderMap(GMaxRHIFeatureLevel)->GetShader<FRayGenTestMS>().GetRayTracingShader());
			PSOInitializer.SetMissShaderTable(RayMissShaderTable);

			FRayTracingPipelineState* PipeLine = PipelineStateCache::GetAndOrCreateRayTracingPipelineState(RHICmdList, PSOInitializer);
			RHICmdList.SetRayTracingMissShader(RHIScene, 0, PipeLine, 0 /* ShaderIndexInPipeline */, 0, nullptr, 0);
			RHICmdList.RayTraceDispatch(PipeLine, RayGenTestRGS.GetRayTracingShader(), RHIScene, GlobalResources, TextureSize.X, TextureSize.Y);
		}
	);

	//Copy shader's output to the render target provided by the client
	RHICmdList.CopyTexture(ShaderOutput->GetRenderTargetItem().ShaderResourceTexture, cachedParams.RenderTarget->GetRenderTargetResource()->TextureRHI, FRHICopyTextureInfo());
	
	//Unbind the previously bound render targets
	GRenderTargetPool.FreeUnusedResource(ShaderOutput);
}
#else // !RHI_RAYTRACING
{
	unimplemented();
}
#endif

output error:

D3D12 ERROR: ID3D12Device::CreateStateObject: Resource bindings for function "RayGen_325ba7afe8c02761" not compatible with associated root signatures (if any): local root signature object: 0x000001759C355090:'Unnamed ID3D12RootSignature Object', global root signature object: 0x00000175AEF8F9A0:'Unnamed ID3D12RootSignature Object'. Error detail: Shader CBV descriptor range (BaseShaderRegister=0, NumDescriptors=1, RegisterSpace=0) is not fully bound in a root signature. If the intent is this will be resolved later when this state object is combined with other state object(s), use a D3D12_STATE_OBJECT_CONFIG subobject with D3D12_STATE_OBJECT_FLAG_ALLOW_LOCAL_DEPENDENCIES_ON_EXTERNAL_DEFINITIONS set in Flags. [ STATE_CREATION ERROR #1194: CREATE_STATE_OBJECT_ERROR]

D3D12: **BREAK** enabled for the previous message, which was: [ ERROR STATE_CREATION #1194: CREATE_STATE_OBJECT_ERROR ]

Exception thrown at 0x00007FFB6A91CD29 (KernelBase.dll) in UnrealEditor.exe: 0x0000087A (parameters: 0x0000000000000001, 0x00000028DC7FBDE0, 0x00000028DC7FDBA0).

Notice I’ve manager to get the TLAS from the ray tracing scene. I’ve also called the TraceRay directly with my own payload struct, that is because I could not get the shader compiler to properly include the headers there, thus making fail to to recognize the FMinimalPayload struct.
I suspect that UE assign its own default callbacks for missing implementation (intersection and anyhit) causing a signature miss-match

I also failed to get the UE TraceRay warpper (TraceVisibilityRay) because that requires me to get FViewInfo on the cpp side, which also giving me a hard time (I was able to create one from the player camera, but that’s not good enough because the ViewUniformBuffer was uninitialized, and I don’t know how to do it)

I would really appreciate if you could share you code, I understand if you don’t wish to.

1 Like

@ZkarmaKun
Ok I have everything set up fine, I think
I’ve made a tutorial about it:

2 Likes

Wonderful exploring!!
Thank you!!