GPU Crash - Invalid use of a render target, resource specified by transition barrier does not match with the state (FParticleStatePosition/FParticleStateVelocity)

Development Rendering
, ,
1

We have been investigating an on-going GPU Crash report that has been affecting a significant portion of players in our game. We’re running a 5.5 custom build, but as far as the renderer specifically is concerned only changes to the shading model paths. What has made this bug so tricky for us is that it is seemingly happening whenever and with no consistent repro.

The crash itself shows either as DXGI_ERROR_DEVICE_REMOVED or DXGI_ERROR_DEVICE_HUNG and doesn’t show us much information in the way of Breadcrumbs only ever showing us a frame number, Aftermath sometimes crashes out and when we are able to get a file it’s without symbols. And DRED hasn’t given us much information either showing DRED: No PageFault data..

We were able to get this information as a result of running with D3DDebug, and as far as my limited understanding goes seems that the data is incorrectly set up for the GPU Driver?

[D3DDebug] ID3D12CommandQueue::ExecuteCommandLists: Using Draw on Command List (0x00000176F9CD5520:'FD3D12CommandList (GPU 0)'): Resource state (0xC0: D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE|D3D12_RESOURCE_STATE_PIXEL_SHADER_RESOURCE) of resource (0x0000017616FC92A0:'FParticleStatePosition') (subresource: 0) is invalid for use as a render target. Expected State Bits (all): 0x4: D3D12_RESOURCE_STATE_RENDER_TARGET, Actual State: 0xC0: D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE|D3D12_RESOURCE_STATE_PIXEL_SHADER_RESOURCE, Missing State: 0x4: D3D12_RESOURCE_STATE_RENDER_TARGET.This has implied to us as a Niagara issue - with Aftermath showing an MMU Fault where a shader instruction is trying to access memory on compute_01 @ 0x00001960. Below is attached the portion relevant of the .log file and an additional nv-gpudmp from a separate unrelated crash.

Investigating a little further has suggested to us that this maybe specifically related to GPU Particles. We’re just unsure if this is related to a specific setting in Niagara, or something at fault with one of our assets.

One other useful thing is that sometimes we do get around roughly which pass it seems to crash at, and it’s a lot of the time

(ID: 0x83ca9b32) [ Active] ContrastAdaptiveShading.

(ID: 0x83ca9b32) [ Active] ContrastAdaptiveShading. We have had this for quite a while now, and have confirmed it’s across _all driver versions_ and has affected everything from a RTX 1650, AMD equivalents and has even crashed out RTX 5090s - Intel GPUs are also not exempt.

Unfortunately, we were unable to get Shader Symbols working and have tried a number of commands, but we dont end up with any symbols.

r.DumpShaderDebugInfo=1
 
r.Shaders.Symbols=1
 
r.Shaders.GenerateSymbols=1
 
r.Shaders.WriteSymbols=1

So we had some questions - Firstly we’re wondering if anyone has spotted an issue similar to this, and if there is potentially already a path that would give us some insight to our crash, and if so - is there something we could cherrypick as a fix?

We’re also wondering how we can get Shader symbols to export correctly in future GPU Crashes, and if there is a performance impact to allowing these on should we want to leave it on to collect more data from crashed players about the specifics.

Any information on how to correctly debug GPU issues going further would also be greatly appreciated - Thanks for reading, I’m hoping you can help us figure it out and learn more about debugging our GPU issues.

2

Steps to Reproduce
Unable to find a solid repro, happens on AMD, Intel and NVIDIA GPUs.

3

Hi there,

I think getting better debug information is the first step here, as there is probably still not enough information to go off to really start looking for a solution to this issue yet. I am not aware of any similar issues

I’m still investigating this issue, but wanted to get something back to you before the end of the day. It looks like the shader symbols not exporting is a know issue in 5.5 that has been resolved in 5.6. See the following issue tracker here. CL 40541074 is the fixing commit. However, it cannot be cherry-picked on its own since it depends on a few other commits. I have attached a git patch file that includes the 3 pre-requisite commits + the fixing commit. I have already resolved any conflicts so this should apply cleanly onto a 5.5 source build. However, keep in mind that this patch should be considered experimental, as I have not tested it extensively. I have at least been able to build and cook content without crashes, and confirmed that shader symbols are now generated under <ProjectDir>/Saved/ShaderSymbols as long as you have the r.Shaders.Symbols=1 CVar set. As long as you do not also set r.Shaders.Optimize=0, exporting shader symbols should not negatively affect the performance of your game. The shader symbols are exported as seperate pdb files, and are not included in the final packaged game content. FYI, you probably also need to set r.GPUCrashDebugging.Aftermath.DumpShaderDebugInfo=1 to get the shader symbols to link up with your aftermath crash dumps as well.

I’m still investigating this, but if you are gathering crash information from a shipping build, you probably need to specify some extra build targets parameters and / or environment variables to get RHI breadcrumb information in a shipping build. It looks like setting bAllowProfileGPUInShipping=true in your *.Target.cs file might be sufficient to get this to work, but I haven’t tested this yet.

Will investigate more tomorrow.

Regards,

Lance Chaney.

4

Some more updates on this issue. I’ve found that Aftermath dumps in 5.5 also don’t have correct shader code association, so crash dumps will fails to show source code currently. The following commit fixes this issue CL 39543979. This commit doesn’t cherry-pick cleanly onto 5.5, and requires quite a bit of manual work to fix merge conflicts. I’ve attached a new patch file for you that should cleanly apply to 5.5 to fix this as well. I’ve tested this on a shipping build using a mangled shader and confirmed I can now get: RHI Breakcrumbs with bAllowProfileGPUInShipping=true, and Aftermath crash dumps, with working shader code association, using r.GPUCrashDebugging.Aftermath.DumpShaderDebugInfo=1 + r.Shaders.Symbols=1.

Final setup for debugging in shipping is as follows:

-- DefaultEngine.ini --

[/Script/Engine.RendererSettings]

r.GPUCrashDebugging.Aftermath.DumpShaderDebugInfo=1

-- Config/Windows/WindowEngine.ini --

[ShaderCompiler]

r.Shaders.Symbols=1

-- *.Target.cs --

bAllowProfileGPUInShipping = true; // Enables RHI Breadcrumbs in shipping

bUseLoggingInShipping = true;

Let me know how you go with this, and update once you’ve collected more debugging information.

Regards,

Lance Chaney

5

Hey Lance! I appreciate the thorough response. We’ll take a look at implementing the fix for Debug Symbols and then get back to you when we have a crash with them. Hope we can figure this out some more!

Thank you!

Alessa.