Page Fault GPU Crash in Lumen Reflections (Hardware Raytracing)

Development Rendering
, ,
1

Hello folks,

I’ve been investigating a GPU crash for a few days now. So far it happens often in Lumen Reflections, but we’ve seen it elsewhere that uses hardware raytracing (e.g. Mega Lights). I’m not sure all the instances have the same root cause, but they are similar enough that I’ll treat them as such for now.

With my current “repro” steps (i.e. playing the game in a specific area until it crashes), the crash happens in a Landscape Closest Hit shader. It seems to fail to reads a constant buffer, or a texture. They are either evicted, destroyed or just untracked by Aftermath.

From what I can understand, there could be two possibles causes for this :

  1. The BLAS points to a valid entry in the SBT, with a valid shader, but the binding (or bindless) resources have been destroyed
  2. The BLAS for some other geometry points to a stale entry in the SBT which happens to be a landscape Hit Shader + Bindings

We’ve also seen a few instances where the crash happens in the internal raytracing shader from the driver. Moreover, we’ve seen this crash across Nvidia and AMD GPUs.

I know there has been quite a lot of changes related to raytracing, SBT, and residency management since 5.7, and we’ve integrated a fews. However, I couldn’t find a CL that was described as fixing this specific issue.

Is any of this sounds familiar and has been fixed by some recent CLs we could integrate?

The closest I could find here was this thread from December : [Content removed]

Thanks!

[Attachment Removed]

2

Steps to Reproduce[Attachment Removed]

3

Hi Olivier,

Do you have any Aftermath crash dumps with the associated shader PDBs on hand that we can take a look at? The info you provided is unfortunately not enough to determine why you are currently crashing with Lumen.

Cheers,

Tim

[Attachment Removed]

4

Hi Olivier,

Sorry for the long wait. I have been trying to dig into the issue a bit more and have a theory as to what might be happening. This is a GPU page fault in your content, not a driver or hardware failure. The dump reports a DMA page fault with a failed virtual address translation on a read, followed by a GPU engine reset. As you have already pointed out, the faulting shader is a ray-tracing closest-hit shader running a landscape material. The exact read that faults is a Virtual Texture page table lookup. The memory being read belongs to a live allocation that the engine evicted from GPU memory, likely due to video memory pressure. Why it was evicted while still in use: With Bindless Resources enabled, a material’s Virtual Texture page table is reached through a bindless descriptor stored inside the material’s uniform buffer. In the ray tracing path, the engine does not register that texture as in use for the frame, so its residency manager treats it as idle and is free to page it out. The next ray dispatch then reads unmapped memory, and the GPU faults.

Three preconditions, all required:

- Bindless Resources enabled

- Virtual Textures used by the material, which landscape materials commonly use

- Video memory pressure is high enough that Unreal’s residency manager begins evicting

What you can try is disabling GPU residency management by setting D3D12.ResidencyManagement=0 in your DefaultEngine.ini to see if the crash happens again. Have you considered upgrading to 5.8 already? We have made significant improvements to GPU residency tracking, but I am not sure if there is a specific CL that will directly fix your issue. We have an open Jira internally with a similar-looking crash, but we have not yet had time to investigate it further.

[Attachment Removed]

5

Hi Olivier,

Ok, then we can rule out the residency management and focus more on the SBT. Assuming you are using a persistent SBT, have you already tried validating it in a development build? In a development build, you can with r.RayTracing.PersistentSBT.ValidateBinding=1, which will add validation data to any binding in the SBT and should assert in case you have any records that differ from what the renderer would write in a given frame. Taking a step back, could you also send me the list of CLs you have cherry-picked from 5.8 that relate to SBT, residency management, etc.? That way, I can verify if there are any CLs you might need to pick up.

Cheers,

Tim

[Attachment Removed]

6

Hi Olivier,

That’s really strange. Yesterday, I found that we have been chasing a similar issue on our end, but we have not had much success nailing down the root cause. You say your QA team can reproduce the issue reliably in a minimal setup. Would it be possible to share that scene with us, so that we can attempt to reproduce the crash? That would be incredibly helpful in finding a root cause. Let me know if that is possible.

[Attachment Removed]

7

Yeah, I figured that would be the case, but I thought it might still be worth asking :slight_smile: If you can share out a build privately, that would be great. Please let me know how you want to proceed with that. Also, do let us know if you find your fix reliable, as we would be happy to upstream the changes if they are not too complex.

[Attachment Removed]

8

Hello,

We found a fix for the page fault in hardware ray tracing lumen reflection.

When binding shader table for ray tracing, if binding is transient, the uniform buffers are not added to DynamicReferencedResources, which update reference count and enable resource residency.

void AddReferencedUniformBuffer(uint32 WorkerIndex, uint32 RecordIndex, uint32 ShaderTableOffset, uint32 InOffsetWithinRootSignature, ERayTracingLocalShaderBindingType BindingType, FD3D12UniformBuffer* UniformBuffer)
{
	if (Lifetime == ERayTracingShaderBindingTableLifetime::Persistent && BindingType == ERayTracingLocalShaderBindingType::Persistent)
	{
		FRecordData& RecordData = HitRecordData[RecordIndex];
 
		for (FRecordUpdateUniformBufferListener* Listener : RecordData.UpdateUniformBufferListeners)
		{
			check(Listener->UniformBuffer != UniformBuffer);
		}
		
		FRecordUpdateUniformBufferListener* Listener = new FRecordUpdateUniformBufferListener(*this);
		Listener->Init(WorkerIndex, UniformBuffer, ShaderTableOffset, RecordIndex, InOffsetWithinRootSignature);
		RecordData.UpdateUniformBufferListeners.Add(Listener);
	}
// FIX BEGIN
#if ENABLE_RESIDENCY_MANAGEMENT
	else if (BindingType == ERayTracingLocalShaderBindingType::Transient || Lifetime != ERayTracingShaderBindingTableLifetime::Persistent)
	{
		FD3D12Resource* UniformBufferResource = UniformBuffer->ResourceLocation.GetResource();
		if (UniformBufferResource)
		{
			AddDynamicReferencedResource(WorkerIndex, UniformBufferResource);
		}
	}
#endif
// FIX END
}

We hit several page fault crashed in the miss shader for light function. The memory of FLightFunctionParametersRayTracing was possiblly evicted or never resident. Those SingleFrame unfiorm buffers are created from FD3D12FastConstantAllocator. If the uniform buffers are used for drawing, that buffer would call UpdateResidency on its UnderlyingResource evetually for every other Uniform buffers in same page. But if all buffers are only used for ray tracing transient binding, then they would not call UpdateResidency.

//create the uniform buffers we need
TUniformBufferRef<FDeferredLightUniformStruct> DeferredLightBuffer = CreateUniformBufferImmediate(GetDeferredLightParameters(View, *LightSceneInfo), EUniformBufferUsage::UniformBuffer_SingleFrame);
TUniformBufferRef<FLightFunctionParametersRayTracing> LightFunctionParameters = CreateLightFunctionParametersBufferRT(LightSceneInfo, View, EUniformBufferUsage::UniformBuffer_SingleFrame);
 
int32 MissIndex = LightAndIndex.Value;
BindLightFunction(RHICmdList, Scene, View, Material, MaterialRenderProxy, DeferredLightBuffer, LightFunctionParameters, MissIndex);

The proposed code fixed a frequent GPU fault related to the lighting function ray tracing when we put ~15 lights with light function in one production map. But it does not occur in the editor, only in the PC package game.

Let me know if you need more information.

Yichen

[Attachment Removed]

9

Hi, thanks for the details. We also received reports from another licensee that these UBs are not added to our residency tracking. I made the dev team aware of your suggested fix so that we can potentially upstream this change. [Content removed] is this the workaround you mentioned?

[Attachment Removed]

10

Yeah, the residency management system needs a few tweaks to properly cover uniform buffer management. This will involve changing more code beyond the fix you already have for the crash you called out, so I can imagine you might be hitting another edge case. For the time being, you might need to keep your workaround in place until we can provide a proper fix

[Attachment Removed]

11

No problem. By the way, I just got word that if you still encounter crashes with residency management turned off, and D3D12.ResourcesStartResident=1, then you might be running into a different issue. In that case, I suggest opening another case so we can discuss details there.

[Attachment Removed]

12

Hi,

That fault does look similar, but to keep the communication focused on one issue, could you please file a separate ticket and link this one to it so I can take a closer look? If you have the shader PDB for the Closest Hit Shader, can you also attach that to the new ticket along with the Aftermath crash artifacts?

Thanks,

Tim

[Attachment Removed]

13

Hello Tim, you can find the aftermath dump attach along with the dxil/pdb/nvdbg files.

For this particular instance it seems like it’s failing to read from a bindless virtual texture.

[Attachment Removed]

14

Hey Tim,

We did integrate a bunch of CL for residency management, but we are not sure we are going to integrate 5.8 at this stage of the project.

I’ll give it a shot with residency management completely turned off to see if that makes a difference, but I get the crash on my machine with 24GB of VRAM. I doubt I hit any video memory pressure. Moreover, we have many other instances of this crash where the page fault resource history list only destroyed resources, or even no resources at all, as if the address points to nothing. I’ve attached one so you can take a look.

[Attachment Removed]

15

Again, reuploading.

[Attachment Removed]

16

Quick update : Disabling residency management did not fix the issue.

[Attachment Removed]

17

Another quick update : r.RayTracing.PersistentSBT=0 seems to reduce the probability of getting the crash, but not completely fix it. Meaning it is still possible to get the crash. I suspect this might just change the timing in the right place.

[Attachment Removed]

18

Hi Tim, yes we always run with ValidateBindings on. Nothing is showing up unfortunately. I’ve also tested with D3DDebug and NV RayTracing Validation, also nothing.

I’ve tried flipping plenty of cvars to try to isolate the issue. The ones that seems to have an effect are :

  • Turning off all dynamic geometry (Landscape and Skeletal Meshes in our cases)
  • Turning off RayTracing Materials (r.RayTracing.EnableMaterials=0)
  • Turning off Persistent SBT

For the SBT, as mentionned earlier, we know it doesn’t fix the issue, but it seems to reduce the probability of the crash. For the other two, I couldn’t repro the crash on my own, but we need a QA pass to make sure since the crash is slippery.

One other interesting point. One of our QA is able to reproduce the issue somewhat consistently (but not always) just by loading into one of our map and not moving at all. There’s not much going on in the raytracing scene at that spot aside from the skeletal mesh of the player. However, I cannot repro this myself on the same build, so it seems like some machine can trigger this crash more than others.

They crashes in a Miss Shader (custom Light Function). This is interesting because we can potentially rull out dynamic geometry as suspect, along with any streaming issue.

[Attachment Removed]

19

Unfortunately the minimal setup is an entire map of the game, but we will try sharing the build through a private channel.

That said, it seems like we might have found a tentative fix. We will do a QA pass and report back.

[Attachment Removed]

20

Yep.

The only remaining doubt we have is that, prior to this fix, we could get the crash to happen by disabling residency management alltogether (D3D12.ResidencyManagement=0). The crash also happens in the editor, where we have D3D12.ResourcesStartResident=1 (set to 0 in the game). So either disabling residency management doesn’t completely disable residency management, or the residency management code path does something that keeps the buffer alive for longer.

It’s also possible that we have multiple causes that all happens to produce a page fault Lumen Reflections.

So we would like your input on this fix given this info.

Thanks!

[Attachment Removed]