Crash in function EnumerateUniformBufferResources in Editor when Raytracing is enabled

Development Asset Creation
, ,
1

Hello,

We hit a crash in Editor when ray tracing is enabled. The callstack is

UE::RHICore::EnumerateUniformBufferResources<...> () UE::RHICore::SetResourcesFromTables<...> () SetRayTracingShaderResources<...> () SetRayTracingHitGroup () ParallelForImpl::ParallelForInternal<...> () FD3D12CommandContext::RHISetBindingsOnShaderBindingTable () FRHICommandSetBindingsOnShaderBindingTable::Execute ()

A texture in the Resource Table became corrupted in development build and null in debug build.

Our investigation suggests that the cached shader data in Scene.CachedRayTracingMeshCommands became invalid when materials are rebuilt if texture asset is updated in texture compiler.

However, adding more debugging code and check seems to be interfering with the bug and makes engine crash in other location, mostly in nanite shader bin binding code, but same cause of texture being invalid.

FYI: UMaterialInstanceDynamic is not used in this situation, just regular UMaterialInstanceConstant. We confirmed that the texture is not GCed by engine.

Any suggestions are welcome!

Thanks,

Yichen Wang

2

Hi there,

Thanks for reaching out. I was able to find some reports on a similar crash, at least from looking at the call stack; however, we do not have a root cause for the crash yet. Would you happen to have a method for us to reproduce this crash on our end?

3

Hi again,

Thanks for providing the repro steps. Which project did you use locally to encounter this crash (not the one with the empty level), and have you made any modifications to the engine that could affect this?

4

Hi Yichen,

Very sorry for the delay in response this week, had a few fires to deal with. This crash you’re highlighting is awfully similar to a crash we’ve literally just resolved this week. Could you please attempt to reproduce with -onethread set and share the full callstack + log?

Also if you could please share the static mesh you mentioned (or store it in a project’s content browser and go to File->Zip up project) and please share it with me, that would be a huge help as I can repro it on my end and see if our changes from this week are the solution…

Thanks,

Jon

5

Hi,

I have not spent any time to verify the crash in stock UE5 version.

I found that the following steps can reproduce the crash relatively easily:

1) Delete the local Zen data cache folder

2) Start the Editor with command line -ddc=noshared

3) Drag a static mesh with multiple material slots using expansive material (in our case, I used a static mesh with 9 material slots, one of them is transparent)

If you still cannot reproduce the crash with steps above, you can create an empty level and fill it with several static mesh actors described in 3 above. Then delete the local Zen DDC cache, launch the editor without shared DDC and open that level.

We have high crash rates in the small feature-testing levels, but not in our production levels.

Yichen

6

Hi,

We encountered this crash in our game project using UE 5.5. We made some code changes in the engine, but nothing related to low-level rendering nor ray tracing.

We investigated the issue last week and made some progress.

When ray tracing is enabled, bindless descriptor heaps will be used for ray tracing. Material will create uniform buffers which are cached by the rendering thread and bind for ray tracing by RHI thread. But rendering thread and RHI thread does not affect the lifetime of uniform buffer by managing its reference count, instead, code will pass the raw pointers around. So, it is critical that material cannot release its uniform buffer until rendering thread finishes the access of uniform buffer.

`void SetShaderMapsOnMaterialResources_RenderThread(FRHICommandListImmediate& RHICmdList, FMaterialsToUpdateMap& MaterialsToUpdate)
{

// Async RDG tasks can call FMaterialShader::SetParameters which touch the material uniform expression cache.
FRDGBuilder::WaitForAsyncExecuteTask();`

But this code is not sufficient for ray tracing, because when binding the resource for ray tracing, the access of resource table and uniform buffers are further delayed to the RHI command running on the RHI thread.

void FRHICommandSetBindingsOnShaderBindingTable::Execute(FRHICommandListBase& CmdList) { INTERNAL_DECORATOR(RHISetBindingsOnShaderBindingTable)(SBT, RayTracingPipelineState, NumBindings, Bindings, BindingType);

The crash is caused by the fact that material uniform buffer is released before RHI command is executed.

When a material or a texture is modified by FShaderCompilingManager or FTextureCompilingManager, game thread will recreate the render state with following code

for (IPrimitiveComponent* AffectedPrimitive : AffectedPrimitives) { AffectedPrimitive->MarkRenderStateDirty(); }In theory, this code will destroy old rendering state and create a new rendering state. The cached ray tracing draw commands will be removed and recreated in the process. But two things are interfering

  1. The recreation is further delayed to the EndOfFrameUpdate in game thread
  2. The update in the rendering thread is not a complete update when component is destroyed and created in same frame

SceneUpdateChangeSetStorage.PrimitiveUpdates.ForEachCommand([&](const FPrimitiveUpdateCommand& Cmd) { // Skip those that were added & removed in the same frame if (Cmd.IsAdd() && !Cmd.IsDelete()) { SceneUpdateChangeSetStorage.AddedPrimitiveSceneInfos.Add(Cmd.GetSceneInfo()); } if (Cmd.IsDelete()) { DeletedPrimitiveSceneInfos.Add(Cmd.GetSceneInfo()); // Skip those that were added & removed in the same frame if (!Cmd.IsAdd()) { SceneUpdateChangeSetStorage.RemovedPrimitiveSceneInfos.Add(Cmd.GetSceneInfo()); SceneUpdateChangeSetStorage.RemovedPrimitiveIds.Add(Cmd.GetPersistentId()); } }

This is what we have found so far. We have not found any workaround for the issue. We may have to disable the ray tracing to prevent the crash in Editor.

Thanks,

Yichen Wang