Summary
NeighborGrid3D data interface: FNDINeighborGrid3DInstanceData_RT::ResizeBuffers() allocates NeighborhoodBuffer / NeighborhoodCountBuffer (BUF_Static) without clearing them. The buffers are only cleared by ResetData (system-reset tick) or by PreStage when the current stage is an output stage for the grid. If a reallocation (SetNumCells / SetMaxNeighborsPerCell changed after the first tick) happens on a non-reset tick while the writing stage is dropped that tick (DirectSet element count 0 → FNiagaraGPUSystemTick skips the stage, so its PreStage clear never runs), a reading stage sees uninitialized memory. A shader loop such as “for (i = 0; i < GetParticleNeighborCount(cell); ++i)” then runs over a garbage count and the GPU hangs (DXGI_ERROR_DEVICE_HUNG, 2 s TDR). Intermittent in the wild (depends on pooled memory contents); made 100% deterministic by filling the count buffer with 0x7FFFFFFF after allocation, and fully fixed by clearing it to 0 there. Code is identical in release-5.8 and ue5-main. Real-world trigger: FluidNinja LIVE 2 (NinjaLiveCore) on activation/re-activation; 10 GPU crashes in one afternoon on our boss map, reproduced on RTX 4070 and RTX 5070 Ti / 5080 (not vendor or driver specific).
Proposed fix (mirrors the existing clears in ResetData / PreStage), in ResizeBuffers after the two Initialize() calls:
AddClearUAVPass(GraphBuilder, NeighborhoodBuffer.GetOrCreateUAV(GraphBuilder), -1);
AddClearUAVPass(GraphBuilder, NeighborhoodCountBuffer.GetOrCreateUAV(GraphBuilder), 0);
Optional hardening: clamp GetParticleNeighborCount’s HLSL return with min(count, MaxNeighborsPerCell), since AddParticle increments the counter before checking the cap.
Aftermath dump (.nv-gpudmp/.nvdbg) with shader symbols available on request: all 50 captured warps are inside the reading stage’s loop. Happy to submit this as a pull request on GitHub if preferred.
What Type of Bug are you experiencing?
Rendering (Graphics / Niagara)
Steps to Reproduce
- Create a GPU emitter with a NeighborGrid3D data interface “Grid” (any resolution, e.g. 16x16x1, MaxNeighborsPerCell 256).
- Add simulation stage A (writer): Iteration Source = Direct Set, Element Count X bound to an int emitter attribute “PointCount”. In the stage, call Grid.AddParticle(0, 0, 0, ExecIndex) for every element.
- Add simulation stage B (reader) after A: Iteration Source = Particles (or any grid). In the stage, read Grid.GetParticleNeighborCount(0) and loop that many times doing trivial work.
- In Emitter Update, set PointCount = 0 for the first few seconds, and call Grid.SetNumCells with a different value on tick 2 or later (e.g. switch from 16x16x1 to 32x32x1 after 0.1 s). Alternatively deactivate and re-activate the component while PointCount is 0 if the system sets the resolution from user parameters after activation.
- Play in editor (D3D12) several times.
Expected Result
Stage B reads a neighbor count of 0 for every cell until stage A has written something. No GPU hang regardless of how the pooled buffer memory was previously used.
Observed Result
Stage A is dropped (element count 0) so its PreStage clear does not run, the reallocated buffers keep whatever the pool contained, and stage B loops over an arbitrary count. Intermittently the GPU exceeds the 2 s TDR limit and the device is removed (DXGI_ERROR_DEVICE_HUNG). With the count buffer filled with 0x7FFFFFFF right after Initialize() in ResizeBuffers the hang is 100% reproducible on every reallocation; with a clear to 0 at the same place it never happens (10+ cycles). Nsight Aftermath with shader symbols shows all captured warps inside stage B’s loop.
Affects Versions
5.8
Platform(s)
Windows
Upload an image
For crash reports, include your callstack
This is a GPU hang (DXGI_ERROR_DEVICE_HUNG, TDR), not a CPU crash. The CPU side is the standard D3D12RHI device-removed handler on RHIInterruptThread (UnrealEditor-D3D12RHI → FGenericCrashContext / “GPU Crash dump Triggered”), so the CPU callstack is not informative.
GPU shader call stack from the Nsight Aftermath dump (shader symbols enabled, UE 5.8, shader hash 9E58A155676AFBF3656034D3083E5148, Niagara stage “DoubleHashPass_2” of FluidNinja NinjaLiveCore):
0 GetVectorByIndex_Emitter_ArrayReader_AttributeUnitPosition __UE_FILENAME_SENTINEL:1268
1 HashGrid_Emitter_Func __UE_FILENAME_SENTINEL:1080 (inside: for(Index = 0; Index < GetParticleNeighborCount_Emitter_NeighborGrid1Output_NeighborCount; ++Index) at line 1053)
2 SimulateMapSimStage4_DoubleHashPass_2 __UE_FILENAME_SENTINEL:1200
3 SimulateMainComputeCS __UE_FILENAME_SENTINEL:1482
All 50 captured warps (Aftermath capture limit) are at lines inside that loop body or in helper functions that are only called from inside it. No warp is outside the loop. The loop bound is GetParticleNeighborCount(NeighborGrid1, cell 0), read from the NeighborhoodCountBuffer that had just been reallocated without a clear.
Active RHI breadcrumbs at the time of the hang: NiagaraGpuComputeDispatch → NiagaraNeighborGrid3DClearNeighborInfo (clears for other instances finished) → NiagaraGPUComputeClearIDToIndexBuffer [Active]. Device state: Hung, no page fault, engine reset occurred.
Additional Notes
How we confirmed the root cause (UE 5.8 source build):
- Added a fill of the NeighborhoodCountBuffer with 0x7FFFFFFF right after the two Initialize() calls in FNDINeighborGrid3DInstanceData_RT::ResizeBuffers. Result: the hang became 100% reproducible on every re-activation of the FluidNinja simulation (2 of 2 attempts, both on the first “off → on” toggle; the very first activation after map load never hangs because the grid is not reallocated then).
- Changed the fill to 0 (and -1 for NeighborhoodBuffer, identical to the existing clears in ResetData / PreStage). Result: no hang in 10+ off/on cycles, no visible change in simulation behavior.
- Without either change the hang was intermittent (about 2 of 3 attempts on an RTX 4070; 9 crashes on RTX 5070 Ti / 5080 across 7 driver versions in one afternoon), i.e. it depends on what the pooled buffer memory previously contained.
Why the regular clears do not cover this case:
- PreStage clears only when Context.IsOutputStage() is true, i.e. before the stage that writes the grid.
- FNiagaraGPUSystemTick drops a DirectSet stage entirely when its bound element count is <= 0, so PreStage (and the clear) is never called for that stage on that tick.
- ResetData only runs on the system-reset tick, but bNeedsRealloc can be set on any later tick when SetNumCells / SetMaxNeighborsPerCell changes a value.
Proposed fix (2 lines in NiagaraDataInterfaceNeighborGrid3D.cpp, ResizeBuffers, after Initialize):
AddClearUAVPass(GraphBuilder, NeighborhoodBuffer.GetOrCreateUAV(GraphBuilder), -1);
AddClearUAVPass(GraphBuilder, NeighborhoodCountBuffer.GetOrCreateUAV(GraphBuilder), 0);
Cost is one clear per reallocation. The same code is present unchanged in release-5.8 and ue5-main.
Optional hardening: the generated HLSL for GetParticleNeighborCount returns the raw counter, which can exceed MaxNeighborsPerCell because AddParticle increments before checking the cap; clamping with min(count, MaxNeighborsPerCell) would bound any loop written against it.
Available on request: .nv-gpudmp + .nvdbg + shader PDB, the decoded Aftermath JSON with per-warp source mapping, and the full UE log with breadcrumbs. Happy to submit the fix as a pull request on GitHub if preferred.