UE5.5.4 - crash in FGPUSortManager::FDynamicValueBuffer::Allocate

Development Platform & Builds
, ,
1

Hello!

We have recently upgraded from UE5.3 to UE5.5.4 and are seeing reports of a crash in FGPUSortManager::FDynamicValueBuffer::Allocate (EXCEPTION_ACCESS_VIOLATION reading address 0x0000000000000028)

Unfortunately we don’t have repro steps, or any real context apart from it seems to happen randomly during gameplay.

I did notice this ticket: [Content removed]

Maybe not exactly the same, but like them we do still use a lot of cascade systems so I did try the fix suggested in there of disabling parallel GDME in FParticleSystemSceneProxy, but we are still seeing instances of the crash with that change in place.

Is this an issue you’re aware of, or do you have any suggestions for fixes / possible causes?

Thanks,

Andy

2

I have just noticed that we have two similar occurrences of this, looks like one from Cascade that I mentioned above and also one from Niagara.

Niagara:

FGPUSortManager::FDynamicValueBuffer::Allocate (GPUSortManager.cpp:326) FGPUSortManager::AddTask (GPUSortManager.cpp:619) FNiagaraGpuComputeDispatch::AddSortedGPUSimulation (NiagaraGpuComputeDispatch.cpp:2269) FNiagaraRendererSprites::GetDynamicMeshElements (NiagaraRendererSprites.cpp:1043) FNiagaraSystemRenderData::GetDynamicMeshElements (NiagaraSystemRenderData.cpp:235) FNiagaraSceneProxy::GetDynamicMeshElements (NiagaraComponent.cpp:522) FDynamicMeshElementContext::GatherDynamicMeshElementsForPrimitive (SceneVisibility.cpp:3999) FDynamicMeshElementContext::LaunchAsyncTask’::2'::<T>::operator() (SceneVisibility.cpp:3971) UE::Tasks::Private::FTaskBase::TryExecuteTask (TaskPrivate.h:504) UE::Tasks::Private::FTaskBase::Init::__l2::<T>::operator() (TaskPrivate.h:185) LowLevelTasks::FTask::Init::__l11::<T>::operator() (Task.h:499) Invoke (Invoke.h:47) LowLevelTasks::TTaskDelegate<T>::TTaskDelegateImpl<T>::Call (TaskDelegate.h:162) LowLevelTasks::TTaskDelegate<T>::TTaskDelegateImpl<T>::CallAndMove (TaskDelegate.h:171) LowLevelTasks::TTaskDelegate<T>::CallAndMove (TaskDelegate.h:308) LowLevelTasks::FTask::ExecuteTask (Task.h:627) LowLevelTasks::FScheduler::ExecuteTask (Scheduler.cpp:277) LowLevelTasks::FScheduler::TryExecuteTaskFrom (Scheduler.cpp:457) LowLevelTasks::FScheduler::WorkerLoop (Scheduler.cpp:514) LowLevelTasks::FScheduler::WorkerMain (Scheduler.cpp:571) LowLevelTasks::FScheduler::CreateWorker’::2'::<T>::operator() (Scheduler.cpp:75) UE::Core::Private::Function::TFunctionRefBase<T>::operator() (Function.h:470) FThreadImpl::Run (Thread.cpp:66) FRunnableThreadWin::Run (WindowsRunnableThread.cpp:156) FRunnableThreadWin::GuardedRun (WindowsRunnableThread.cpp:79) kernel32 +0x001259c BaseThreadInitThunk ntdll +0x005af77 RtlUserThreadStart

Cascade:

FGPUSortManager::FDynamicValueBuffer::Allocate (GPUSortManager.cpp:326) FGPUSortManager::AddTask (GPUSortManager.cpp:619) FFXSystem::AddSortedGPUSimulation (ParticleGpuSimulation.cpp:4617) FGPUSpriteDynamicEmitterData::GetDynamicMeshElementsEmitter (ParticleGpuSimulation.cpp:3031) FParticleSystemSceneProxy::GetDynamicMeshElements (ParticleSystemRender.cpp:6628) FDynamicMeshElementContext::GatherDynamicMeshElementsForPrimitive (SceneVisibility.cpp:3999) FDynamicMeshElementContext::LaunchRenderThreadTask'::2’::::operator() (SceneVisibility.cpp:3950)
UE::Core::Private::Function::TFunctionRefBase::operator() (Function.h:470)
TFunctionGraphTaskImpl::DoTaskImpl (TaskGraphInterfaces.h:1710)
TFunctionGraphTaskImpl::DoTask (TaskGraphInterfaces.h:1703)
TGraphTask::ExecuteTask (TaskGraphInterfaces.h:634)
UE::Tasks::Private::FTaskBase::TryExecuteTask (TaskPrivate.h:504)
FBaseGraphTask::Execute (TaskGraphInterfaces.h:482)
FNamedTaskThread::ProcessTasksNamedThread (TaskGraph.cpp:779)
FNamedTaskThread::ProcessTasksUntilQuit (TaskGraph.cpp:667)
FTaskGraphCompatibilityImplementation::ProcessThreadUntilRequestReturn (TaskGraph.cpp:1453)
FTaskGraphCompatibilityImplementation::WaitUntilTasksComplete (TaskGraph.cpp:1526)
FTaskGraphInterface::WaitUntilTaskCompletes (TaskGraphInterfaces.h:401)
FBaseGraphTask::Wait (TaskGraphInterfaces.h:539)
FVisibilityTaskData::ProcessRenderThreadTasks (SceneVisibility.cpp:4754)
FDeferredShadingSceneRenderer::BeginInitViews (SceneVisibility.cpp:5778)
FDeferredShadingSceneRenderer::Render (DeferredShadingRenderer.cpp:1648)
RenderViewFamilies_RenderThread (SceneRendering.cpp:5455)
FRendererModule::BeginRenderingViewFamilies’::95'::<T>::operator() (SceneRendering.cpp:5758) UE::Core::Private::Function::TFunctionRefBase<T>::operator() (Function.h:470) FRenderThreadCommandPipe::EnqueueAndLaunch'::5’::::operator() (RenderingThread.cpp:1538)
UE::Core::Private::Function::TFunctionRefBase::operator() (Function.h:470)
TFunctionGraphTaskImpl::DoTaskImpl (TaskGraphInterfaces.h:1710)
TFunctionGraphTaskImpl::DoTask (TaskGraphInterfaces.h:1703)
TGraphTask::ExecuteTask (TaskGraphInterfaces.h:634)
UE::Tasks::Private::FTaskBase::TryExecuteTask (TaskPrivate.h:504)
FBaseGraphTask::Execute (TaskGraphInterfaces.h:482)
FNamedTaskThread::ProcessTasksNamedThread (TaskGraph.cpp:779)
FNamedTaskThread::ProcessTasksUntilQuit (TaskGraph.cpp:667)
RenderingThreadMain (RenderingThread.cpp:317)
FRenderingThread::Run (RenderingThread.cpp:442)
FRunnableThreadWin::Run (WindowsRunnableThread.cpp:156)
FRunnableThreadWin::GuardedRun (WindowsRunnableThread.cpp:79)
kernel32
BaseThreadInitThunk
RtlUserThreadStart`

3

Hi,

Did the crash still occur when you disable parallel GDME for both Niagara & Cascade?

One change that does spring to mind is 40656300 “Fix race with Post Init Views particle systems”, I think it’s a long shot for what you are seeing but worth testing.

Thanks,

Stu

4

So I think I see an issue here, we do run a mix of Cascade & Niagara in FN, but we generally do not allow sorting.

There’s a mutex that is individual to Cascade & Niagara so they can both enter the GPU Sort Manager at the same time. I’ll need to confirm this but if you wanted to test yourself you could add a mutex inside FGPUSortManager::AddTask.

Cheers,

Stu

5

Thanks for testing that out, you still need the external lock when adding the sort info to the array. I’m debating changing the API for this or just having two locks.

Thanks,

Stu

6

We did only disable parallel GDME for Cascade, and not Niagara. I can try disabling it for both if you think that is worth trying.

I will also cherry-pick that change. Would you recommend both having that fix and the disabled parallel GDME at the same time?

In case this doesn’t fix it, do you have any idea what situation might trigger this so we can maybe try track it down/verify the fixes because right now we only see this in larger scale playtesting but have no reproduction steps.

Thanks,

Andy

7

Ahh yes, I see. I will try out adding a mutex in there.

I suppose disabling parallel GDME for both Cascade and Niagara could achieve the same thing, if that all then happens on the same thread.

Thanks,

Andy

8

Yeah disable both should do the same thing, what I’ll likely do is shift that mutex into the gpu sort manager and use that as the mutex from outside.

Not 100% until I get a repro for it.

Thanks,

Stu

9

I was just able to repro this in editor by taking a Cascade and a Niagara system that needed sorting, and spamming instances of them in the same scene and I could fairly quickly reproduce the crash.

I can also confirm that having one shared mutex in the GPUSortManager does seem to fix it for me (I just locked within FGPUSortManager::AddTask)

Thanks,

Andy