GPU Crash in TranslucentLightInjectionShaders

We’re getting intermittent crashes from devs running our packaged game on PC and in PIE reported to our Sentry crash report system. This occurs when running with Epic Scalability settings and has not been seen when running at High/Medium/Low quality settings.

We have Nvidia Aftermath crash dumps enabled and using Nsight Graphics I can see that almost all the crashes are coming from the same shader (TranslucentLightInjectionShaders), and occur in the same place in ShadowGetPhysicalPage.

MMU Fault Error during a GPU memory Read of a destroyed unnamed resource 
 or other resource(s) at address 0x00000024b9c5d000 in this shader location:
Fragment Shader fragment_01 @ 0x00000b00 
 
A shader instruction caused an MMU fault when accessing memory.
This can be caused by shader bugs and binding setup issues.

Can you give any advice as to the cause of this hang? We have made no custom modifications to the shader.

[Attachment Removed]

Hi there,

Would you be able to send through the aftermath crash dump (*.nv-gpudmp file), and associated dxil and pbd shader files. You should be able to find the dxil / pdb files required by clicking the Show Symbol Files button in the debug window:

[Image Removed]

Could I also get some more information about your rendering setup, including: All r.* rendering settings from your project defaultEngine.ini, and an overview of your lighting setup in the level(s) that are crashing (including any light functions used on any lights, how many shadow casting lights you have, what type of lights you are using, any modified settings on your lights from the default, etc).

Edit: Also might be relevant to know if you are rendering any extra views (extra scene cameras, planar reflections, stereo or split screen rendering, etc).

Edit2: It looks like your VSM page table texture might be getting destroyed before it is read in the pixel shader here. Can you run your project with the -rhivalidation argument and let me know if you get any RHI validation errors in your log. Especially if they relate to VSM buffers / textures.

Regards,

Lance Chaney

[Attachment Removed]

I’ve attached all the relevant aftermath crash dump files and listed all the r.* rendering settings set in DefaultEngine.ini below:

r.MobileHDR = False
r.MobileNumDynamicPointLights = 4
r.MobileDynamicPointLightsUseStaticBranch = False
r.Mobile.EnableStaticAndCSMShadowReceivers = False
r.DiscardUnusedQuality = False
r.AllowOcclusionQueries = True
r.hzbocclusion = False
r.MinScreenRadiusForLights = 0.030000
r.MinScreenRadiusForDepthPrepass = 0.030000
r.MinScreenRadiusForCSMDepth = 0.010000
r.PrecomputedVisibilityWarning = False
r.TextureStreaming = True
Compat.UseDXT5NormalMaps = False
r.ClearCoatNormal = True
r.ReflectionCaptureResolution = 128
r.ForwardShading = False
r.AllowStaticLighting = False
r.NormalMapsForStaticLighting = False
r.GenerateMeshDistanceFields = True
r.GenerateLandscapeGIData = False
r.TessellationAdaptivePixelsPerTriangle = 48.000000
r.SeparateTranslucency = True
r.TranslucentSortPolicy = 0
TranslucentSortAxis = (X=0.000000,Y=-1.000000,Z=0.000000)
r.CustomDepth = 3
r.DefaultFeature.Bloom = True
r.DefaultFeature.AmbientOcclusion = True
r.DefaultFeature.AmbientOcclusionStaticFraction = False
r.DefaultFeature.AutoExposure = False
r.DefaultFeature.AutoExposure.ExtendDefaultLuminanceRange = True
r.DefaultFeature.AutoExposure.Method = 0
r.DefaultFeature.MotionBlur = False
r.DefaultFeature.LensFlare = False
r.DefaultFeature.AntiAliasing = 2
r.StencilForLODDither = False
r.EarlyZPass = 3
r.EarlyZPassMovable = False
r.DBuffer = True
r.ClearSceneMethod = 1
r.SelectiveBasePassOutputs = False
r.AllowGlobalClipPlane = False
r.GBufferFormat = 1
r.MorphTarget.Mode = True
vr.InstancedStereo = False
vr.MultiView = False
r.WireframeCullThreshold = 5.000000
r.SupportStationarySkylight = False
r.SupportLowQualityLightmaps = False
r.SupportPointLightWholeSceneShadows = True
r.SupportAtmosphericFog = True
UIScaleRule = ShortestSide
UIScaleCurve = (EditorCurveData=(PreInfinityExtrap=RCCE_Constant,PostInfinityExtrap=RCCE_Constant,Keys=,DefaultValue=340282346638528859811704183484516925440.000000),ExternalCurve=None)
r.SupportMaterialLayers = True
r.SSS.Checkerboard = 1
rhi.SyncInterval = 1
r.GTSyncType = 2
r.OneFrameThreadLag = 1
r.VSync = 0
rhi.SyncSlackMS = 0
r.VirtualTextures = True
r.VirtualTexturedLightmaps = False
r.DistanceFieldBuild.Compress = True
r.DynamicGlobalIlluminationMethod = 1
bDefaultParticleCutouts = True
r.SupportSkyAtmosphereAffectsHeightFog = False
r.Shadow.CSMCaching = True
r.SkinCache.CompileShaders = True
r.BufferVisualizationOverviewTargets = BaseColor,Metallic,SubsurfaceColor,WorldNormal,SeparateTranslucencyRGB,FinalImage,TuquePBRColorThreshold,WorldTangent,SeparateTranslucencyA,TuquePBRRoughnessThreshold,TuquePBRSpecularThreshold,Opacity,SceneDepth,Roughness,Specular,ShadingModel,,SceneDepthWorldUnits,SceneColor,PreTonemapHDRColor,PostTonemapHDRColor
r.ReflectionMethod = 1
r.Shadow.Virtual.Enable = 1
bEnableVirtualTextureOpacityMask = True
r.GPUSkin.Support16BitBoneIndex = True
r.GPUSkin.UnlimitedBoneInfluences = True
SkeletalMesh.UseExperimentalChunking = 1
r.AntiAliasingMethod = 4
r.CustomDepthTemporalAAJitter = True
r.VT.TileSize = 128
r.OptimizedWPO = 1
r.Nanite.AllowTessellation = 1
r.Nanite.Tessellation = 1
r.nanite.dicingrate = 5.5
landscape.Nanite.MarchingSquaresVisibility = False
r.Lumen.ScreenProbeGather.ShortRangeAO = 0
r.Lumen.ScreenProbeGather.ShortRangeAO.ScreenSpace.FoliageOcclusionStrength = 1.5
r.Lumen.DiffuseIndirect.SSAO = 1
r.HeterogeneousVolumes.IndirectLighting = 1
r.LocalFogVolume.GlobalStartDistance = 20
r.SkipDrawOnPSOPrecaching = 1
r.FilmGrain = 0
r.Water.SingleLayer.ShadersSupportVSMFiltering = 1
r.Water.SingleLayer.VSMFiltering = 0
r.RayTracing = True
r.Lumen.HardwareRayTracing = True
r.Lumen.DiffuseIndirect.SSAO = 1
foliage.DisableWPOInRaytracing = 1
r.RayTracing.Culling.Radius = 15000
r.RayTracing.AsyncBuild=True
r.RayTracing.Culling.Angle = 0
fx.Niagara.AsyncGpuTrace.HWRayTraceEnabled = True
r.Lumen.Reflections.HardwareRayTracing.Translucent.Refraction.EnableForProject = False
bSkinnedMeshComponentsAreVisibleInRayTracingByDefault = False
bStaticMeshComponentsAreVisibleInRayTracingByDefault = False
r.SkinCache.DefaultBehavior = 0
r.Material.DefaultAutoMaterialUsage=False
rhi.Bindless.Resources=Enabled
r.AOGlobalDistanceField=True
r.ContactShadows=False
r.EyeAdaptation.BlackHistogramBucketInfluence=1

I’m still waiting to hear back from our lighting artist re. info on the lighting setup.

As far as I’m aware, there are no extra views.

I’ve also got a few users running with -rhivalidation and I’ll report back if it flags anything.

[Attachment Removed]

I’m not able to reproduce this so far, even with your provided settings.

Some sanity checks:

- Are you changing any VSM settings at runtime by any chance?

- Do you see the following message in your logs at all `Recreating Shadow.Virtual.PhysicalPagePool due to size or flags change. This will also drop any cached pages.`

- Do you have any custom scalability settings that might change the settings for Epic scalability

Unfortunately I’m not sure how much more help I can be without being able to reproduce the issue, or getting some more indication of where the issue might be coming from. Did you manage to get any RHI validation logs from computers that have crashed?

Regards,

Lance Chaney

[Attachment Removed]

Hi there,

I’m temporarily taking over this case from Lance until he’s back. Thanks a lot for the update, that’s a good find. It would be interesting to know if this guard improves stability.

Best regards,

Sam

[Attachment Removed]

Hello,

This case has been re-assigned for a dev to look into when available. Some additional information that may help us here:

What is the latest driver version you have seen this crash on? I ask because the one in the crash dump is a year old now, and it’d be good to ensure it’s not specific to a range of driver versions.

It also looks like bindless resourcess are enabled (rhi.Bindless.Resources=Enabled) , do you know what that is being used for in the project?

[Attachment Removed]

Have you tried disabling BindlessResourceHeap or changing the syncpoint of the BindlessResourceHeap recyle from EnqueueEndOfPipeTask to FD3D12Adapter::FrameFence? We encountered a similar problem with CSM and bindless enabled. Since we were just cherry-picking some code from the mainstream, I’m not sure whether this is a UE problem or ours.

[Attachment Removed]

Hi,

The first thing to point out is that there’s literally nothing a shader can legally do to cause a MMU crash. If we were looking at a timeout, then shader changes could be reasonable to try and work around the hang, but not in this case.

So this has to come from the RHI (resource management) or driver (more unlikely, but not unheard of).

Since you have enabled bindless which is experimental, that does seem like a very likely culprit - it is not in widespread use and would entail changes to RHI resource management. Seems to me the next step should be to turn off bindless resources and see if the crash goes away.

In either case, as has been requested above - an isolated repro would also be very valuable, in particular if it turns out to be caused by bindless, so we can get to the bottom of this.

Best,

Ola

[Attachment Removed]

We have solved this GPU hung using some workaround, but still dont know the exact reason.

If you have any discover related to this problem​, plz let us know.

Thanks.

[Attachment Removed]

Hi,

As I stated above the likely problem is use of bindless, this is not production ready so some issues are to be expected. Switching to minimal bindless (5.7) may help as this is the path receiving more focus recently. I’m not aware of a specific suggestion for what may cause this issue or any known workaround at the moment and so can only recommend turning off bindless if the issue persists with minimal bindless.

Best,

Ola

[Attachment Removed]

Hi folks,

We are introducing a change in 5.8.2 that may have a positive impact on this crash and a known issue with flickering translucency. Note, the commit message likely means to say:

- replaced: inner batched -> inner unbatched -> outer batched -> outer unbatched

- with: inner batched -> outer batched -> inner unbatched -> outer unbatched

This change also moves every cascade’s UAV->RTV transition ahead of the raster injection passes instead of interleaving them per cascade.

UE6 CL#56603393 (2e588c) [Lit Translucency] Reorder batched/unbatched lights injection into TLV.

- replaced: inner batched -> inner unbatched -> outer batched -> outer batched

- with: inner batched -> outer batched -> inner unbatched -> outer batched

- this seems to avoid what seems to be a synchronization bug that caused lights on translucency to flicker while the root cause is investigated.

UE 5.8 CL#56702016 (30b4cd)

[Attachment Removed]

Can you also send through the associated *.dxil and *.pdb files? Otherwise I can’t view the shader source in Nsight.

Regards,

Lance Chaney

[Attachment Removed]

Hi Lance,

Apologies, it would only let me attach 1 file to each reply so I’ve edited the OG message to add the remaining files. Should be both the .dxil and .pdb files as well as the .nvdbg.

Cheers

Jon

[Attachment Removed]

  • We’re not changing VSM settings at runtime
  • We’re not seeing “Recreating Shadow.Virtual.PhysicalPagePool” in the logs
  • This crash occurs with standard UE Epic scalability settings

We get some RHI validation warnings but nothing related to VSMs or shadows. We’re trying our best to narrow this down and or find a 100% repro case. I’ll update you when I have more info.

Thanks for taking the time to help us

[Attachment Removed]

Just had another GPU crash occur in Editor (not PIE) while moving stuff around in a test gym. Scalability was only set to Medium too.

The aftermath dump shows the same MMU error reading memory in the same function (ShadowGetPhysicalPage) in the same shader as before (TranslucentLightInjectionShaders.dxil).

We had -rhivalidation enabled and the only error in the logs is shown below. The crash occurred ~40mins later.

Error: === Handled ensure: ===
Ensure condition failed: Bindings[Layout.StaticSlot] == nullptr  [File:Z:\ba\w\501660219612b7b9\UnrealEngine\Engine\Source\Runtime\RHI\Private\RHIValidation.cpp] [Line: 1468] 
Uniform buffer 'FViewUniformShaderParameters' was bound statically and is now being bound on a specific RHI shader. Only one binding model should be used at a time.
    From    => [Z:\ba\w\501660219612b7b9\UnrealEngine\Engine\Source\Runtime\RHI\Private\RHIValidation.cpp(1468)]

I’ve attached the relevant logs containing the aftermath and breadcrumb data too.

[Attachment Removed]

Hmm, the rhi validation error is probably not relevant then.

One thing I note is that you seem to have multiple directional lights in the scene. Which is a pretty non-standard setup.

Gym__ScriptedEvents.DirectionalLight1

Gym__ScriptedEvents.DirectionalLight2

You might want to try ensuring you only ever have 1 directional light and see if you continue to get the crash, or whether crashes only happen on maps with multiple directional lights.

You might want to copy the light setup from that map, and your render settings, into a blank project and do some kind of soak test to see if you can get it to crash.

The other lights that were rendering when the crash occurred, according to your logs were:

BP_SE__EmoDungeonBrazier-2

BP_SE__EmoDungeonBrazier-5

BP_SE__EmoDungeonBrazier-4

BP_SE__EmoDungeonBrazier-3

So you could try copying those ,and the directional lights, into a blank project to test with.

If we can get a reproduction project for this, then we can create a bug report, even if the reproduction rate isn’t 100%.

Regards,

Lance Chaney.

[Attachment Removed]

We may have spotted the problem.

We’ve noticed that VSM sampling is guarded in InjectBatchMainCS but not in InjectMainPS.

We’ve added a guard check as shown below

if (VirtualShadowMapId != INDEX_NONE)
{
	FVirtualShadowMapSampleResult VirtualShadowMapSample = SampleVirtualShadowMapTranslatedWorld(VirtualShadowMapId, TranslatedWorldPositionForLighting);
	ShadowFactor *= VirtualShadowMapSample.ShadowFactor;
}

We’ll let you know if we still get reported GPU crashes

[Attachment Removed]

So far we’ve seen less crash instances but still seeing a few.

We’ve found a potential NAN issue that we’ve fixed in InjectMainPS and InjectBatchMainCS if the LightVector is 0,0,0

Original Code:    float3 NormalizedLightVector = normalize(LightVector)
New Code:           float3 NormalizedLightVector = SafeNormalize(LightVector)
New function: 
// Utility: safe normalize that returns 0 if length is ~0
float3 SafeNormalize(in float3 v, float eps = 1e-8f)
{
	float len2 = dot(v, v);
	if (len2 > eps)
	{
		return v * rsqrt(len2);
	}
	else
	{
		return float3(0, 0, 0);
	}
}

We’ll keep you posted

[Attachment Removed]

Thanks for that.

Are you getting crashes in the same places? or have the crashes changed since you made these modifications?

FYI, I believe the reason there is a guard on the compute shader InjectBatchMainCS, is because this batches over multiple lights, but not all of these lights necessarily cast shadows, so there needs to be a runtime check. The pixel shader version runs on 1 light, so correct shader variant (VIRTUAL_SHADOW_MAP flag) should be selected depending on whether the light casts shadows or not, without requiring a runtime check.

Regards,

Lance Chaney

[Attachment Removed]

Unfortunately we’ve had issues with our Sentry logging (missing Aftermath dumps) to I’ve not been able to verify the cause.

This has now been fixed and the potential NAN shader fix is in too.

We’ve not had any new GPU hangs in the past 36 hrs. Fingers crossed the NAN issue was the cause.

I’ll keep you updated.

Thanks for the info on the reason for the guard in InjectBatchMainCS. That all makes sense.

[Attachment Removed]