We’re getting intermittent crashes from devs running our packaged game on PC and in PIE reported to our Sentry crash report system. This occurs when running with Epic Scalability settings and has not been seen when running at High/Medium/Low quality settings.
We have Nvidia Aftermath crash dumps enabled and using Nsight Graphics I can see that almost all the crashes are coming from the same shader (TranslucentLightInjectionShaders), and occur in the same place in ShadowGetPhysicalPage.
MMU Fault Error during a GPU memory Read of a destroyed unnamed resource
or other resource(s) at address 0x00000024b9c5d000 in this shader location:
Fragment Shader fragment_01 @ 0x00000b00
A shader instruction caused an MMU fault when accessing memory.
This can be caused by shader bugs and binding setup issues.
Can you give any advice as to the cause of this hang? We have made no custom modifications to the shader.
Would you be able to send through the aftermath crash dump (*.nv-gpudmp file), and associated dxil and pbd shader files. You should be able to find the dxil / pdb files required by clicking the Show Symbol Files button in the debug window:
[Image Removed]
Could I also get some more information about your rendering setup, including: All r.* rendering settings from your project defaultEngine.ini, and an overview of your lighting setup in the level(s) that are crashing (including any light functions used on any lights, how many shadow casting lights you have, what type of lights you are using, any modified settings on your lights from the default, etc).
Edit: Also might be relevant to know if you are rendering any extra views (extra scene cameras, planar reflections, stereo or split screen rendering, etc).
Edit2: It looks like your VSM page table texture might be getting destroyed before it is read in the pixel shader here. Can you run your project with the -rhivalidation argument and let me know if you get any RHI validation errors in your log. Especially if they relate to VSM buffers / textures.
I’m not able to reproduce this so far, even with your provided settings.
Some sanity checks:
- Are you changing any VSM settings at runtime by any chance?
- Do you see the following message in your logs at all `Recreating Shadow.Virtual.PhysicalPagePool due to size or flags change. This will also drop any cached pages.`
- Do you have any custom scalability settings that might change the settings for Epic scalability
Unfortunately I’m not sure how much more help I can be without being able to reproduce the issue, or getting some more indication of where the issue might be coming from. Did you manage to get any RHI validation logs from computers that have crashed?
I’m temporarily taking over this case from Lance until he’s back. Thanks a lot for the update, that’s a good find. It would be interesting to know if this guard improves stability.
This case has been re-assigned for a dev to look into when available. Some additional information that may help us here:
What is the latest driver version you have seen this crash on? I ask because the one in the crash dump is a year old now, and it’d be good to ensure it’s not specific to a range of driver versions.
It also looks like bindless resourcess are enabled (rhi.Bindless.Resources=Enabled) , do you know what that is being used for in the project?
Have you tried disabling BindlessResourceHeap or changing the syncpoint of the BindlessResourceHeap recyle from EnqueueEndOfPipeTask to FD3D12Adapter::FrameFence? We encountered a similar problem with CSM and bindless enabled. Since we were just cherry-picking some code from the mainstream, I’m not sure whether this is a UE problem or ours.
The first thing to point out is that there’s literally nothing a shader can legally do to cause a MMU crash. If we were looking at a timeout, then shader changes could be reasonable to try and work around the hang, but not in this case.
So this has to come from the RHI (resource management) or driver (more unlikely, but not unheard of).
Since you have enabled bindless which is experimental, that does seem like a very likely culprit - it is not in widespread use and would entail changes to RHI resource management. Seems to me the next step should be to turn off bindless resources and see if the crash goes away.
In either case, as has been requested above - an isolated repro would also be very valuable, in particular if it turns out to be caused by bindless, so we can get to the bottom of this.
As I stated above the likely problem is use of bindless, this is not production ready so some issues are to be expected. Switching to minimal bindless (5.7) may help as this is the path receiving more focus recently. I’m not aware of a specific suggestion for what may cause this issue or any known workaround at the moment and so can only recommend turning off bindless if the issue persists with minimal bindless.
We are introducing a change in 5.8.2 that may have a positive impact on this crash and a known issue with flickering translucency. Note, the commit message likely means to say:
Apologies, it would only let me attach 1 file to each reply so I’ve edited the OG message to add the remaining files. Should be both the .dxil and .pdb files as well as the .nvdbg.
We’re not seeing “Recreating Shadow.Virtual.PhysicalPagePool” in the logs
This crash occurs with standard UE Epic scalability settings
We get some RHI validation warnings but nothing related to VSMs or shadows. We’re trying our best to narrow this down and or find a 100% repro case. I’ll update you when I have more info.
Just had another GPU crash occur in Editor (not PIE) while moving stuff around in a test gym. Scalability was only set to Medium too.
The aftermath dump shows the same MMU error reading memory in the same function (ShadowGetPhysicalPage) in the same shader as before (TranslucentLightInjectionShaders.dxil).
We had -rhivalidation enabled and the only error in the logs is shown below. The crash occurred ~40mins later.
Error: === Handled ensure: ===
Ensure condition failed: Bindings[Layout.StaticSlot] == nullptr [File:Z:\ba\w\501660219612b7b9\UnrealEngine\Engine\Source\Runtime\RHI\Private\RHIValidation.cpp] [Line: 1468]
Uniform buffer 'FViewUniformShaderParameters' was bound statically and is now being bound on a specific RHI shader. Only one binding model should be used at a time.
From => [Z:\ba\w\501660219612b7b9\UnrealEngine\Engine\Source\Runtime\RHI\Private\RHIValidation.cpp(1468)]
I’ve attached the relevant logs containing the aftermath and breadcrumb data too.
Hmm, the rhi validation error is probably not relevant then.
One thing I note is that you seem to have multiple directional lights in the scene. Which is a pretty non-standard setup.
Gym__ScriptedEvents.DirectionalLight1
Gym__ScriptedEvents.DirectionalLight2
You might want to try ensuring you only ever have 1 directional light and see if you continue to get the crash, or whether crashes only happen on maps with multiple directional lights.
You might want to copy the light setup from that map, and your render settings, into a blank project and do some kind of soak test to see if you can get it to crash.
The other lights that were rendering when the crash occurred, according to your logs were:
BP_SE__EmoDungeonBrazier-2
BP_SE__EmoDungeonBrazier-5
BP_SE__EmoDungeonBrazier-4
BP_SE__EmoDungeonBrazier-3
So you could try copying those ,and the directional lights, into a blank project to test with.
If we can get a reproduction project for this, then we can create a bug report, even if the reproduction rate isn’t 100%.
Are you getting crashes in the same places? or have the crashes changed since you made these modifications?
FYI, I believe the reason there is a guard on the compute shader InjectBatchMainCS, is because this batches over multiple lights, but not all of these lights necessarily cast shadows, so there needs to be a runtime check. The pixel shader version runs on 1 light, so correct shader variant (VIRTUAL_SHADOW_MAP flag) should be selected depending on whether the light casts shadows or not, without requiring a runtime check.