Checking through your log, the crash reports a device removed error, the typical “DXGI_ERROR_DEVICE_HUNG”, usually tied with a GPU running out of memory and hanging. But, the end of the report shows that the operation was nowhere close to run out of memory:
Video Memory Stats from frame ID 4692: Local Budget: 10259.00 MB Local Used: 6604.09 MB System Budget: 48293.12 MB System Used: 344.95 MB
So, the issue’s cause is not about memory, something else is generating the crash. And looking further at the log, I think this is the key line:
“Fault Description”: “A shader instruction caused an MMU fault when accessing memory.\nThis can be caused by shader bugs and binding setup issues.”, “Fault Name”: “MMU Fault Error”, “Shader GPU PC Address”: “ray_tracing_02 @ 0x0001cad0”, “Shader mapping”: null
This error shows that the GPU’s memory is trying to access an element that is either invalid or missing. If we consider this along with the heavy use of procedural geo in your scene, I think it can boil down to this:
Your procedural geo generates/updates it’s form
UE registers that geo in memory with Ray Tracing
The geo updates again, or is culled, and a section of memory is cleared
Shader “ray_tracing_02” looks for the old memory generated at the start, which is no longer there
The GPU throws a “AddressTranslationError” (or Page Fault), and the whole thing crashes
Now, in order to resolve this, there are a few things to test:
Since the issue is most likely tied to how your procedural geo interacts with Ray Tracing, then it’s worth testing to disable said interaction. Select your geo actors on scene, check their Details panel, look for “Visible in Ray Tracing”, and uncheck that option
In case your entire scene uses procedural geo, and it’s not easy to isolate, then you can force the project to rely on Software Ray Tracing instead. To do that, go to Project Settings > Engine > Rendering, look for the Hardware Ray Tracing section, and disable “Support Hardware Ray Tracing”
Finally, if possible, tinker and adjust the frequency in which your procedural geo updates, in order to give the engine more time to read the memory entries, before they change
This is a voxel / minecraft-ish game, so the whole scene is procedural. But I’m not changing them frequently, just adding new ones.
I’m also not updating the geometry in existing, active procedural components, but I am pooling the components. So when the camera moves, it could be removing components from one side of the active world, then changing the geometry and displaying them on the other side of the world in the same frame.
I will introduce a frame delay in the pooling strategy to delay the time between the last time a component was rendered and the time when I reset its geometry.
Okay, there was a bug in my implementation of the fix. Waiting a frame between when the mesh is removed from the scene and when it gets modified has fixed this issue. Let the songs be written.
I set r.RayTracing.LightFunction=0 (on Claude’s request), and instead of a GPU crash I hit an ensure at line 1031 of RayTracing.cpp saying that the instance is being skipped because the geometry is evicted.
After doing stat RayTracingGeometry, It looks like some kind of spike doubles the amount of Resident memory early in a run. Requested memory doesn’t change. There is a full second hitch, and when it comes back Resident memory is huge.
Edit: This is not correct. If I am requesting a lot more than is available (because I’ve deliberately set r.RayTracing.ResidentGeometryMemoryPoolSizeInMB too low), the Resident memory will occasionally spike way beyond what I’ve set the limit to be.
That spike then evicts geometry, some of which is still in use. I think evicting stuff that’s in use is the bug, but it’s possible something is running away and allocating too much memory.