GPU Hang with failed PSOs compile

Development Rendering
, ,
1

Hello,

We are trying to solve hangs happening on certain user machines.

The problem at the moment is that we don’t have enough meaningful info to figure out what would be causing the hang.

We only have few bits in logs and they are suggesting some failed PSOs compilation happening.

Are you encoutering this, do you have some recommendations on how to diagnose the issue?

Is there any way we could rule out PSOs?

We were also wondering if PSO compile putting pressure on the game could cause a hang as well?

Best,

Daniel Quach

[Attachment Removed]

2

Hi there!

The PSO1 log has very little info, including what GPU and driver version were in use and no breadcrumbs or aftermath dump - however it is possible both are the result of a stale or corrupt PSO cache. The PSO0 log which crashed with DeviceRemovedNoGpuFault on a 5090 with driver version 591.86, might a driver bug - possibly the shader heap overflow issue that is fixed in 595.x and had similar symptoms e.g. a crash with very little information. It looks like that one output a .nv-gpudmp file which we can look at If you attach it. I recommend requesting the user update to the latest driver. You can also set your suggested driver version to a 595.xx driver in your <project>/config/DefaultHardware.ini

[GPU_NVIDIA Windows]
SuggestedDriverVersion="595.36"

If you have reports of these GPU crashes on newer driver versions then that would point to this being more likely a PSO cache issue.

Is there any way we could rule out PSOs?

It’s possible for a corrupted PSO cache to cause crashes, but we’ve never had a live repro. The way we’ve reproduced this artificially is by writing random bytes to mapped memory, which can result in a GPU crash rarely, and then on subsequent loads of the game it will crash. The crash will go away when the PSO cache is cleared. So, in theory this could occur if you had a memory stomp that poisoned the PSO cache. The resulting GPU crash from this has no DRED, no Aftermath, no active shaders or resources, similar to the PSO0.log

If you have a machine it repros on, you can ask the user to manually move their PSO cache elsewhere so it gets regenerated, and if that fixes the issue, you can look at the PSO cache to try and identify from the hash and the contents of a good and bad cache if there’s a difference and possibly whether the PSO is corrupt or stale - we’ve never had a live repro for this so it’s theoretical. You may also be able to look for other crash reports from that user and see if the oldest reported crash looks like a memory stomp.

You can also make it easier for users to clear their PSO cache by exposing the -clearPSODriverCache functionality.

We were also wondering if PSO compile putting pressure on the game could cause a hang as well?

The only known hangs in this area have come from the driver trying to optimize shaders at some time after PSO compilation and when you have a ton of PSOs that optimization could cause a shader heap overflow which has been fixed in the 595 branch.

Are you not seeing breadcrumbs in your logs with crash reports? If not, do you have r.GPUCrashDebugging.Breadcrumbs enabled in your builds? You may also want to enable r.D3D12.LightweightDRED which will incur a slight overhead but may be worth it while you are hunting down these issues with a limited audience.

[Attachment Removed]

3

Thanks for the insights!

Good shout about the driver version, we will set it as the base one.

About breadcrumbs, we do have them but it’s not showing on certain logs, the only clues we have is around PSOs

We’ve added more aftermath info and will update you if once we get a meaningful crash

[Attachment Removed]

4

Hello,

I’ve uncovered some shaders that are actually crashing validation layer.

Those shaders are getting recorded in PSO bundle with a permutation WAVESIZE(64)

Seems the crash is GPU dependent.

I managed to reproduce this crash on a RTX 5070

LogD3D12RHI: Error: [D3DDebug] ID3D12Device::CreateComputeShader: Shader wants wave lane count of 64 but maximum wave lane count for device is 32. To check for support, check device caps via the CheckFeatureSupport() API

Global/FTranscodePageToGPU_CS/2:/Engine/Private/Nanite/NaniteTranscode.usf|TranscodePageToGPU

Global/FLumenCardBatchDirectLightingCS/0:/Engine/Private/Lumen/LumenSceneDirectLighting.usf|LumenCardBatchDirectLightingCS

[Attachment Removed]

5

Congrats on the launch!

Sorry to hear the wrong WaveSize is being selected though - can you confirm if the wavesize(32) variation is not in your bundle or if it is in the bundle but it isn’t being selected correctly in SelectTranscodeWaveSize() in NaniteStreamingManager.cpp?

You can also try setting r.Nanite.Streaming.TranscodeWaveSize=32 and see if that fixes the validation error on the 5070.

Did you record PSOs on a 50 series? It’s possible that if you didn’t that you didn’t ever generate those permutations. Also, I was under the impression you were using PSO precaching - are you using a mix of bundling and precaching?

Support for WaveSize32 for LumenCardBatchDirectLightingCS in LumenSceneDirectLighting was added in CL#51544453 (c4b6e4) [Lumen scene direct lighting] [Lumen scene direct lighting] Add an option to cull lights to card tiles using one group per tile instead of one thread per tile. The new option is the new default. Based on test results, the new option seems always no slower and is ~0.2 ms faster (without async) when there is ~1.2k lights in the scene

UE 5.8 CL#52603973 (3a484e)

However, it looks like WaveSize 64 shouldn’t have been selected even before that change, which makes it seem more likely that the other permutations were not compiled into the bundle?

if (GRHISupportsWaveOperations && RHISupportsWaveOperations(Views[0].GetShaderPlatform()))
    {
        // 64 wave size is preferred for FLumenCardBatchDirectLightingCS
        if (GRHIMinimumWaveSize <= 64 && GRHIMaximumWaveSize >= 64)
        {
            WaveOpWaveSize = 64;
        }
        else if (GRHIMinimumWaveSize <= 32 && GRHIMaximumWaveSize >= 32)
        {
            // TODO: wave32 support
            // WaveOpWaveSize = 32;
        }
    }

[Attachment Removed]

6

Thank you Alex!

I’ve put a check on SelectTranscodeWaveSize(), it’s always correct.

We have r.PSOPrecache.GlobalShaders 1

I’m suspecting it’s gathering all permutations and so during runtime, it tries to compile this permutation but fails on a non supported GPU

In PSO bundle load code, it just does a straight load + compile without checking whether the permutation is valid or not

[Attachment Removed]

7

My colleague indicated this was fixed for compute PSOs to not be fatal for bundled PSOs in 5.7 with this changelist:

CL#43263054 (c721a5) Cleanup used precached PSOs, when they are kept in memory to workaround driver cache inefficiencies.

Once a PSO is needed for rendering, the PSO precache system sets a flag that marks the precached PSO (if existing) as used, and enqueues it for cleanup if it is currently being held in memory. This is enabled with r.PSOPrecache.KeepInMemoryUntilUsed=2.

Also remove some unnecessary duplicated code now that we have compute initializers, and clearly separate the implementation for the compile task between graphics and compute. This clean up work should be a “no fuctional change” cleanup but comes with opportunities to simplify the code further down the line.

Can you try integrating and testing with that?

[Attachment Removed]

8

Hello Alex,

Apologies for the reply delay.

I’ve integrated the CL and tested it, it doesn’t fix the issue.

We can still store permutations in bundle (spc) that are invalid for runtime.

I’ve actually managed to get more info:

IMO there’s a problem with ShouldPrecacheWithWaveSize test

We build/capture PSO bundle on a machine that has GRHIGlobals.MaximumWaveSize = 64 (e.g. machine having a 7900 XTX) => It gets stored in the bundle we distribute and load on runtime

We run the game with this bundle on a machine that has GRHIGlobals.MaximumWaveSize = 32 (e.g. machine having a 5070 RTX) => PSO gets loaded, created without any check and triggers validation issue.

Shaders using it are TSR, Nanite and Lumen

	inline bool ShouldPrecacheWithWaveSize(const FShaderPermutationParameters& Parameters, int32 WaveSize)
	{
		if (WaveSize)
		{
			if (WaveSize < GRHIGlobals.MinimumWaveSize || WaveSize > GRHIGlobals.MaximumWaveSize)
			{
				return false;
			}
		}
 
		return true;
	}

The fix I’m testing is to only allow Precache for Wavesize 0, 32 (which should be standard?)

And we take the hit on missing PSO above these wavesizes.

On our side, we’re still being hit by GPU crashes and wanna remove this issue as potential culprit

[Attachment Removed]

9

Thanks for the update! A wavesize of 32 is fairly standard aside from some AMD and Intel GPUs. I’m sending this over to my colleague who is more familiar with the PSO bundling situation in case he has further suggestions about how to handle the wave sizes.

[Attachment Removed]

10

Hi,

are these PSOs coming from bundles PSO caches or PSO precaching?

We indeed still have issues with wave size on PSO compilation from both these code paths but slightly different setup. For bundled they can get recorded and not compile on the actual device, for precaching all compute permutations can be compiled at booth without checking the wave size supports. But compilation of these PSOs shouldn’t be fatal but just log - it’s not ideal but shouldn’t crash at least. Do you have a callstack from when it fails? Ideally without async PSO compilation set so the callstack is clear where the PSO compilation crash comes from.

Cheers,

Kenzo

[Attachment Removed]