New random crash in FD3D12GPUFence::Poll since project migration to UE 5.7.4

Hi Epic Support Team!

Since our project migrated from UE5.5 to UE 5.7.4 we get a random crash in FD3D12GPUFence::Poll() EXCEPTION_ACCESS_VIOLATION reading address 0x0000000000000028, also in build with array size checks we get this this assert in the same call stack. In Github I see that GPUMessaging.cpp haven’t changed a lot in a while, but this is something we didn’t hit before.

Is this a known issue and is there a fix for it?

[ 00 ] RaiseException ( KERNELBASE.dll )
[ 01 ] ReportAssert(wchar_t const *,void *) ( WindowsPlatformCrashContext.cpp:2414 )
[ 02 ] FWindowsErrorOutputDevice::Serialize(wchar_t const *,ELogVerbosity::Type,FName const &) ( WindowsErrorOutputDevice.cpp:85 )
[ 03 ] FOutputDevice::LogfImpl::__l3::<lambda_1>::operator()(wchar_t *) ( OutputDevice.cpp:81 )
[ 04 ] GrowableLogfV(wchar_t const *,char *,FOutputDevice::LogfImpl::__l3::<lambda_1> const &) ( VarargsHelper.h:47 )
[ 05 ] FOutputDevice::LogfImpl(wchar_t const *,...) ( OutputDevice.cpp:81 )
[ 06 ] FOutputDevice::Logf(wchar_t const[98] &,char const *,char const *,int,wchar_t *) ( OutputDevice.h:246 )
[ 07 ] AssertFailedImplV(char const *,char const *,int,void *,wchar_t const *,char *) ( AssertionMacros.cpp:177 )
[ 08 ] FDebug::CheckVerifyFailedImpl2V(char const *,char const *,int,wchar_t const *,char *) ( AssertionMacros.cpp:722 )
[ 09 ] FDebug::CheckVerifyFailedImpl2(char const *,char const *,int,wchar_t const *,...) ( AssertionMacros.cpp:745 )
[ 10 ] TArray<TRefCountPtr<FD3D12SyncPoint>,TSizedInlineAllocator<8,32,TSizedDefaultAllocator<32> > >::RangeCheck(int) ( Array.h:1095 )
[ 11 ] TArray<TRefCountPtr<FD3D12SyncPoint>,TSizedInlineAllocator<8,32,TSizedDefaultAllocator<32> > >::operator[](int) ( Array.h:1173 )
[ 12 ] TArray<TRefCountPtr<FD3D12SyncPoint>,TSizedInlineAllocator<8,32,TSizedDefaultAllocator<32> > >::operator[](int) ( Array.h:1186 )
[ 13 ] FD3D12GPUFence::Poll(FRHIGPUMask) ( D3D12DirectCommandListManager.cpp:31 )
[ 14 ] FRHIGPUMemoryReadback::IsReady() ( RHIGPUReadback.h:34 )
[ 15 ] GPUMessage::FSystem::PollMessages() ( GPUMessaging.cpp:137 )
[ 16 ] GPUMessage::FSystem::BeginMessageScope(FRDGBuilder &) ( GPUMessaging.cpp:79 )
[ 17 ] GPUMessage::FScope::FScope(FRDGBuilder &) ( GPUMessaging.cpp:286 )
[ 18 ] FDeferredShadingSceneRenderer::Render(FRDGBuilder &,FSceneRenderUpdateInputs const *) ( DeferredShadingRenderer.cpp:1851 )
[ 19 ] RenderViewFamily_RenderThread(FRDGBuilder &,FSceneRenderer *,FSceneRenderUpdateInputs const *) ( SceneRendering.cpp:4937 )
[ 20 ] FRendererModule::BeginRenderingViewFamilies::__l115::<lambda_4>::operator()(FRDGBuilder &,FSceneRenderFunctionInputs const &) ( SceneRendering.cpp:5232 )
[ 21 ] `FRendererModule::BeginRenderingViewFamilies'::`115'::<lambda_4>::<lambda_invoker_cdecl>(FRDGBuilder &,FSceneRenderFunctionInputs const &) ( SceneRendering.cpp:5234 )
[ 22 ] UE::Core::Private::Function::TFunctionRefBase<UE::Core::Private::Function::TFunctionStorage<1>,bool __cdecl(FRDGBuilder &,FSceneRenderFunctionInputs const &)>::operator()(FRDGBuilder &,FSceneRenderFunctionInputs const &) ( Function.h:414 )
[ 23 ] `FSceneRenderProcessor::Execute'::`30'::<lambda_4>::operator()(FRHICommandListImmediate &) ( SceneRenderBuilder.cpp:894 )
[ 24 ] FCycleCounter::Stop() ( StatsSystemTypes.h:1599 )
[ 25 ] FScopeCycleCounter::{dtor}() ( StatsSystemTypes.h:1659 )
[ 26 ] ExecuteCommand(FRHICommandList *,TVariant<TUniqueFunction<void __cdecl(void)>,TUniqueFunction<void __cdecl(FRHICommandList &)>,TUniqueFunction<void __cdecl(FRHICommandListImmediate &)> > const &,FRenderCommandTag const &) ( RenderingThread.cpp:1540 )
[ 27 ] FRenderThreadCommandPipe::ExecuteCommands::__l2::<lambda_1>::operator()(TVariant<TUniqueFunction<void __cdecl(void)>,TUniqueFunction<void __cdecl(FRHICommandList &)>,TUniqueFunction<void __cdecl(FRHICommandListImmediate &)> > &&,FRenderCommandTag const &) ( RenderingThread.cpp:1937 )
[ 28 ] UE::RenderCommandPipe::FCommandList::ConsumeCommands<`FRenderThreadCommandPipe::ExecuteCommands'::`2'::<lambda_1> >(FRenderThreadCommandPipe::ExecuteCommands::__l2::<lambda_1> const &) ( RenderingThread.h:431 )
[ 29 ] FRenderThreadCommandPipe::ExecuteCommands(UE::RenderCommandPipe::FCommandList &) ( RenderingThread.cpp:1935 )
[ 30 ] `FRenderThreadCommandPipe::EnqueueAndLaunch'::`5'::<lambda_1>::operator()() ( RenderingThread.cpp:1924 )
[ 31 ] UE::Core::Private::Function::TFunctionRefBase<UE::Core::Private::Function::TFunctionStorage<1>,void __cdecl(void)>::operator()() ( Function.h:414 )
[ 32 ] TFunctionGraphTaskImpl<void __cdecl(void),0>::DoTaskImpl(TUniqueFunction<void __cdecl(void)> &,ENamedThreads::Type,TRefCountPtr<FBaseGraphTask> const &) ( TaskGraphInterfaces.h:1101 )
[ 33 ] TFunctionGraphTaskImpl<void __cdecl(void),0>::DoTask(ENamedThreads::Type,TRefCountPtr<FBaseGraphTask> const &) ( TaskGraphInterfaces.h:1094 )
[ 34 ] TGraphTask<TFunctionGraphTaskImpl<void (void),0> >::ExecuteTask() ( TaskGraphInterfaces.h:697 )
[ 35 ] std::_Atomic_storage<int,4>::load(std::memory_order const) ( atomic:932 )
[ 36 ] AtomicLoadRelaxed(int volatile *) ( Atomic.h:62 )
[ 37 ] UE::Trace::FChannel::IsEnabled() ( Channel.inl:18 )
[ 38 ] UE::Trace::FChannel::operator bool() ( Channel.inl:24 )
[ 39 ] Finished(unsigned __int64) ( TaskTrace.cpp:199 )
[ 40 ] TaskTrace::FTaskTimingEventScope::{dtor}() ( TaskTrace.cpp:322 )
[ 41 ] UE::Tasks::Private::FTaskBase::TryExecuteTask() ( TaskPrivate.h:519 )
[ 42 ] FBaseGraphTask::Execute(TArray<FBaseGraphTask *,TSizedDefaultAllocator<32> > &,ENamedThreads::Type,bool) ( TaskGraphInterfaces.h:495 )
[ 43 ] FNamedTaskThread::ProcessTasksNamedThread(int,bool) ( TaskGraph.cpp:791 )
[ 44 ] FNamedTaskThread::ProcessTasksUntilQuit(int) ( TaskGraph.cpp:680 )
[ 45 ] RenderingThreadMain(FEvent *) ( RenderingThread.cpp:261 )
[ 46 ] FRenderingThread::Run() ( RenderingThread.cpp:393 )
appError called: Assertion failed: (Index >= 0) & (Index < ArrayNum) [File:D:\HordeAgent\GZW_UE57\Sync\Engine\Source\Runtime\Core\Public\Containers\Array.h] [Line: 1095] 
Array index out of bounds: 0 into an array of size 0

[Attachment Removed]

Steps to Reproduce
Crash appears randomly on DX12 window game client. Repro rate is ~1/100.

[Attachment Removed]

Hi Oleksii.

It looks like that crash is caused by the RDG deleting the GPU readback fence for the GPU breadcrumbs while it is still in use by the render thread (see RHIGPUReadback.cpp:40). This might be happening because in 5.6 we made the execution of the RDG commands asynchronous without any awaits (r.RDG.ParallelExecute=2) at the end of RDGBuilder::Execute(), which was not the default in 5.5. You could try setting r.RDG.ParallelExecute=1 to get back the old behavior at the cost of having a wait at the end of RDGBuilder::Execute for the RDG task to complete to see if that clears up the assert. Since you have such a low repro rate, you would need to decide if that is worth it to you. Let me know what you think.

[Attachment Removed]