Crash when PSO precache + cache occurs while Insights tracing

Development Rendering
, ,
1

This question was created in reference to: [Crash when compiling bundled PSOs with Insights [Content removed]

Hi,

We are on UE5.6 (so it contains CL 41715756) however we are experiencing this same crash only while insights tracing is enabled.

The repro is to launch a Development build with `-tracefile=trace` and the crash occurs on the first frame (or quickly following) when PSO caching is resumed (and PSO precache is enabled).

Note: PSO cache startup mode is 0, and pso precache is disabled in the .ini. They are manually resumed / enabled through CVar once the loading screen appears. The crash started occurring after doing that change to our code.

Thanks in advance,

Hugo

2

Hi Hugo,

Thanks for reaching out. Just to confirm, do you also encounter this crash when running your development build with the RHI breadcrumbs support compiled out (WITH_RHI_BREADCRUMBS 0)? If you can run your build without the breadcrumb code, this can indicate whether or not the CL you pointed out actually fixed the PSO compilation race condition. Would you happen to have more information about the crash, such as log statements, dump files, etc.? Unfortunately, that call stack you provided does not reveal much about why the crash occurred.

3

Hi Hugo,

Okay, if you can repro this in a sample project, that would be hugely helpful. Please let me know once you have this available.

4

Hi Hugo,

It looks like that change could be useful. Would you mind trying it out and reporting back here if it fixes your crash? I could then look into getting this merged into our codebase.

5

Ok, thanks for letting me know. I let the team know that the fix should be considered for acceptance, as you were to confirm that it works for you. Our team is still reviewing the PR, but I hope to have an update on the PR by the end of next week. Please let me know if you have any further questions, though.

6

Hi Tim,

Tweaking WITH_RHI_BREADCRUMBS in Engine\Source\Runtime\RHI\Public\RHIBreadcrumbs.h to the following

#ifndef WITH_RHI_BREADCRUMBS
#define WITH_RHI_BREADCRUMBS 0
#endif

unfortunately I encounter a lot of compilation errors. Locally we have fixed the crash like so if that helps:

FGraphicsPipelineState* PipelineStateCache::GetAndOrCreateGraphicsPipelineState(FRHICommandList& RHICmdList, const FGraphicsPipelineStateInitializer& Initializer, EApplyRendertargetOption ApplyFlags)
{
...
		FPSOCompilationDebugData PSOCompilationDebugData;
#if WITH_RHI_BREADCRUMBS
		if (DoAsyncCompile && false) // <---
		{
			PSOCompilationDebugData.BreadcrumbRoot = FRHIBreadcrumbNode::GetNonNullRoot(RHICmdList.GetCurrentBreadcrumbRef());
			PSOCompilationDebugData.BreadcrumbNode = PSOCompilationDebugData.BreadcrumbRoot
				? RHICmdList.GetCurrentBreadcrumbRef()
				: nullptr;
		}
#endif // WITH_RHI_BREADCRUMBS
...
}

and the same for PipelineStateCache::GetAndOrCreateComputePipelineState().

I will try to repro in a simpler project so hopefully I can send a sample project.

Best,

Hugo

7

Hi Tim,

I unfortunately couldn’t reply in Lyra with a similar setup, however I stumbled this PR: https://github.com/EpicGames/UnrealEngine/pull/13754

Does it make sense to you ?

Thanks,

Hugo

8

Hi Tim,

I can confirm the crash is gone with that change applied (replacing the ` && false` we had instead).

Best,

Hugo