PCG deadlock due lack of brackground workers

Development Platform & Builds
, ,
1

This question was created in reference to: [PCG using all background workers and causing a [Content removed]

Also related to [Content removed] .

Hi,

We’re seeing similar deadlocks as the ones reported in the previous thread when using runtime PCG generation.

As suggested, we started to tweak pcg.MaxPercentageOfThreadsToUse and pcg.MaxPercentageOfExecutingThreads.

Something that I noticed is that when tweaking those values in a conservative way I was still able to have deadlocks. When looking at the parallel callstack I was seeing PCG using all the of the Background Workers available for my particular hardware.

FPCGGraphExecutor::ExecuteScheduling(…) does some math with those CVars, and eventually calculate the required workers as follows:

  // Number of threads to use for PCG Element execution (outside of main thread tick)
  const int32 MaxExecutingThreads = PCGGraphExecutor::CVarGraphMultithreading.GetValueOnAnyThread() ? FMath::Max(0, (int32)(MaxNumThreads * MaxPercentageOfExecutingThreads)) : 0;

  // Number of threads to use for PCG Async operations 
  const int32 MaxPCGAsyncThreads = FMath::Min(MaxNumThreads - MaxExecutingThreads, CVarMaxNumTasks.GetValueOnAnyThread());

But even with those restriction ( and tweaks in the mentioned CVars ) I still see PCG in general using all the possible Background Workers.

Could it be possible that some PCG tasks are sent to the Background Worker pool without honoring the heuristic from FPCGGraphExecutor ?

Thanks,

Alvaro

2

Hi Alvaro,

Did you have a repro step on how/where this is happening?

To give you some context, one of the original cases where we ended up with a deadlock on the bulk data (on the PCG landscape cache) had to do with the way the loading the data was blocking and expecting something else to complete before going on, which led to the aforementioned deadlock.

I’ve dispatched to my colleague for a deeper look into it in the meantime.

Cheers,

Julien

3

Hey Alvaro,

The current PCG Graph Executor does in fact not respect the total number of tasks that can be started by the PCGAsync api, it does instead give a number based on number of available threads. (but it shouldn’t really matter and shouldn’t lead to stalls/deadlocks)

We’ve tested cases with very low amounts of worker threads (even 0) and currently don’t have a known use case where we can repro what you are seeing.

What I would be intersted to know is what do you describe as an actual deadlock? What task is not advancing and why? Does it have an external dependency and keep ownership of the worker thread? Like Julien mentioned we had issues where our tasks were keeping hold of the worker thread starving the system but those have been fixed since.

Another thing we’ve fixed after 5.7 release is that we’ve replaced all native mutexes by non native ones which fixes some issues with runtime gen where we were creating to many native primitives and busting the platform’s budget.

I would probably need more details on the stall to give more insight.

Cheers,

Patrick

4

Hey Alvaro, this OpenReadBulkData bug has been fixed to my knowledge a while ago with CL 40273823 in UE5/Main.

Also we’ve changed the behavior of the LandscapeCache so that it can only be prepared on the main thread as this was causing other issues accessing its textures. I see the case is opened for 5.6 but both those issues were fixed in 5.6.

Do you guys run a vanilla engine?

Cheers,

Patrick

5

Hi [mention removed]​

,

We don’t have an isolated way to reproduce this issue.

We’re currently being very aggressive with runtime PCG, which is maybe why it’s not a very common deadlock.

Something that we noticed was that we can have a more consistent repro rate if we set PCG.FrameTime to 1.

On our end, as a temporary fix, we ended up adding a CriticalSection owned by FPCGGraphExecutor and propagated through the FPCGAsyncState of the task we’ll execute in the workers. So, when FPCGAsync::AsyncProcessing(…) tries to execute background tasks it won’t overlap requests from other threads executing FPCGGraphExecutor::ExecuteScheduling(…).

We noticed some times FPCGGraphExecutor::ExecuteScheduling(…) was queuing task ( ie, calling FPCGAsync::AsyncProcessing(…) ) from different threads without honoring the quota imposed. Adding the mutex seems to alleviate the problem.

Regards,

Alvaro

6

Hi [mention removed]​ ,

"

What I would be intersted to know is what do you describe as an actual deadlock? What task is not advancing and why? Does it have an external dependency and keep ownership of the worker thread? Like Julien mentioned we had issues where our tasks were keeping hold of the worker thread starving the system but those have been fixed since.

"

We’re experiencing basically the same as described in this thread: [Content removed]

ie, FPCGLandscapeCacheEntry::TouchAndLoad(…) stuck in a critical section and while other thread ( the one that acquired the critical section ) is waiting for OpenReadBulkData() to get the data, but this needs background workers which are not available on our case ( since PCG is using all of them and most of them are locked in a mutex ).

I could share a parallel stack in case it helps.

"

Another thing we’ve fixed after 5.7 release is that we’ve replaced all native mutexes by non native ones which fixes some issues with runtime gen where we were creating to many native primitives and busting the platform’s budget.

"

We adopted some of those changes since we were having cashes in PS5 because we exhausted the system resources for mutexes.

I thought that won’t be a problem for Windows mutexes, but let me know if it’s not the case.

Thanks in advance,

Alvaro

7

Hi [@Patrick [Content removed] (Epic Games)​ ,

This thread was closed, but I’m not sure if you might still need the the parallel callstack in case you want to take a look on your side ( see me previous reply ).

Please let me know.

Regards,

Alvaro

8

We’re seeing this exact situation on 5.7.4. Running with runtime PCG hitting a deadlock and we have several threads stuck inside FPCGLandscapeCacheEntry::TouchAndLoad with the callstack like:

[Inline Frame] BWGame-Win64-Test.exe!Windows::EnterCriticalSection(Windows::CRITICAL_SECTION *) Line 238 C++ Symbols loaded.
[Inline Frame] BWGame-Win64-Test.exe!UE::FWindowsRecursiveMutex::Lock() Line 43 C++ Symbols loaded.
[Inline Frame] BWGame-Win64-Test.exe!UE::TScopeLockUE::FWindowsRecursiveMutex::{ctor}(UE::FWindowsRecursiveMutex &) Line 28 C++ Symbols loaded.
[Inline Frame] BWGame-Win64-Test.exe!FScopeLock::{ctor}(UE::FWindowsRecursiveMutex *) Line 148 C++ Symbols loaded.
[Inline Frame] BWGame-Win64-Test.exe!FPCGLandscapeCacheEntry::TouchAndLoad(int) Line 331 C++ Symbols loaded.
[Inline Frame] BWGame-Win64-Test.exe!UPCGLandscapeCache::GetCacheEntryInternal(AActor *) Line 959 C++ Symbols loaded.
[Inline Frame] BWGame-Win64-Test.exe!UPCGLandscapeCache::GetCacheEntryInternal(const ULandscapeInfo *) Line 860 C++ Symbols loaded.
BWGame-Win64-Test.exe!UPCGLandscapeCache::GetCacheEntry(const ULandscapeInfo * InLandscapeInfo, const UE::Math::TIntPoint & InComponentKey, ALandscapeProxy * InLandscapeProxy, bool bInAllowLoadOrCreate) Line 846 C++ Symbols loaded.
BWGame-Win64-Test.exe!UPCGLandscapeCache::SampleMetadataOnPoint(ALandscapeProxy * Landscape, const UE::Math::TTransform & InTransform, __int64 & OutMetadataEntry, UPCGMetadata * OutMetadata) Line 1000 C++ Symbols loaded.
BWGame-Win64-Test.exe!PCGWorldQueryHelpers::ApplyRayHitMetadata(const FHitResult & HitResult, const FPCGWorldRaycastQueryParams & QueryParams, const UE::Math::TVector & RayDirection, const UE::Math::TTransform & InTransform, __int64 & OutMetadataEntry, UPCGMetadata * OutMetadata, TWeakObjectPtr<UWorld,FWeakObjectPtr> World) Line 337 C++ Symbols loaded.
BWGame-Win64-Test.exe!UPCGWorldRayHitData::SamplePoint(const UE::Math::TTransform & InTransform, const UE::Math::TBox & InBounds, FPCGPoint & OutPoint, UPCGMetadata * OutMetadata) Line 322 C++ Symbols loaded.
BWGame-Win64-Test.exe!UPCGSpatialData::ProjectPoint(const UE::Math::TTransform & InTransform, const UE::Math::TBox & InBounds, const FPCGProjectionParams & InParams, FPCGPoint & OutPoint, UPCGMetadata * OutMetadata) Line 280 C++ Symbols loaded.
BWGame-Win64-Test.exe!PCGSurfaceSampler::SampleSurface::__l2::<lambda_2>::operator()(int StartReadIndex, int StartWriteIndex, int Count) Line 312 C++ Symbols loaded.

and several others made it past this lock and are blocked in OpenReadBulkData

BWGame-Win64-DebugGame.exe!FEventWin::Wait(unsigned int WaitTime, const bool bIgnoreThreadIdleStats) Line 1929 C++ Symbols loaded.
[Inline Frame] BWGame-Win64-DebugGame.exe!FEvent::Wait() Line 76 C++ Symbols loaded.

BWGame-Win64-DebugGame.exe!UE::BulkData::Private::OpenReadBulkData(const UE::BulkData::Private::FBulkMetaData & BulkMeta, const FIoChunkId & BulkChunkId, __int64 Offset, __int64 Size, EAsyncIOPriorityAndFlags Priority, TFunction<void __cdecl(FArchive &)> && Read) Line 386 C++ Symbols loaded.
BWGame-Win64-DebugGame.exe!FBulkData::TryLoadDataIntoMemory(FIoBuffer Dest) Line 1676 C++ Symbols loaded.
BWGame-Win64-DebugGame.exe!FBulkData::GetCopy(void * * Dest, bool bDiscardInternalCopy) Line 940 C++ Symbols loaded.
BWGame-Win64-DebugGame.exe!FPCGLandscapeCacheEntry::SerializeFromBulkData() Line 423 C++ Symbols loaded.
[Inline Frame] BWGame-Win64-DebugGame.exe!FPCGLandscapeCacheEntry::TouchAndLoad(int) Line 334 C++ Symbols loaded.
[Inline Frame] BWGame-Win64-DebugGame.exe!UPCGLandscapeCache::GetCacheEntryInternal(AActor *) Line 959 C++ Symbols loaded.
[Inline Frame] BWGame-Win64-DebugGame.exe!UPCGLandscapeCache::GetCacheEntryInternal(const ULandscapeInfo *) Line 860 C++ Symbols loaded.
[Inline Frame] BWGame-Win64-DebugGame.exe!UPCGLandscapeCache::GetCacheEntry(const ULandscapeInfo *) Line 846 C++ Symbols loaded.
BWGame-Win64-DebugGame.exe!UPCGLandscapeCache::SampleMetadataOnPoint(ALandscapeProxy * Landscape, const UE::Math::TTransform & InTransform, __int64 & OutMetadataEntry, UPCGMetadata * OutMetadata) Line 998 C++ Symbols loaded.
BWGame-Win64-DebugGame.exe!PCGWorldQueryHelpers::ApplyRayHitMetadata(const FHitResult & HitResult, const FPCGWorldRaycastQueryParams & QueryParams, const UE::Math::TVector & RayDirection, const UE::Math::TTransform & InTransform, __int64 & OutMetadataEntry, UPCGMetadata * OutMetadata, TWeakObjectPtr<UWorld,FWeakObjectPtr> World) Line 337 C++ Symbols loaded.
BWGame-Win64-DebugGame.exe!UPCGWorldRayHitData::SamplePoint(const UE::Math::TTransform & InTransform, const UE::Math::TBox & InBounds, FPCGPoint & OutPoint, UPCGMetadata * OutMetadata) Line 322 C++ Symbols loaded.
BWGame-Win64-DebugGame.exe!UPCGSpatialData::ProjectPoint(const UE::Math::TTransform & InTransform, const UE::Math::TBox & InBounds, const FPCGProjectionParams & InParams, FPCGPoint & OutPoint, UPCGMetadata * OutMetadata) Line 280 C++ Symbols loaded.
BWGame-Win64-DebugGame.exe!PCGSurfaceSampler::SampleSurface::__l2::<lambda_2>::operator()(int StartReadIndex, int StartWriteIndex, int Count) Line 312 C++ Symbols loaded.
[Inline Frame] BWGame-Win64-DebugGame.exe!UE::Core::Private::Function::TFunctionRefBase<UE::Core::Private::Function::FFunctionRefStoragePolicy,int __cdecl(int,int,int)>::operator()(int <Params_0>, int <Params_1>, int <Params_2>) Line 414 C++ Symbols loaded.
[Inline Frame] BWGame-Win64-DebugGame.exe!FPCGAsync::Private::AsyncProcessing::__l53::(const int) Line 585 C++ Symbols loaded.
BWGame-Win64-DebugGame.exe!FPCGAsync::Private::AsyncProcessing::__l55::<lambda_4>::operator()() Line 610 C++ Symbols loaded.

9

We also have the same problem with PCG randomely getting stuck in OpenReadBulkData in UE 5.7.1 compiled from source. When this happen, we still have some background thread free to do other work. But checking the Dispatcher inside the Batch variable, there is no request left to do in it (WaitingRequests Head and Tail are null), and the IODispatcher thread is doing nothing, as is the event completion was not called(?) which lead to Event->Wait() indefinitely waiting.
We are still investigating to see when and how this happen, it’s very random to reproduce, any tips would be welcome.

Edit :

After more search, it looks like our background threads where full. I mistook them with the background threads Standby, which look not used (even if some where also locked by PCG). And the problem looks coming from a build with a compressed pak, where the IoDispatcher use the FFileIoStore, which was pushing some ScatterBlockDecompressionTask. And since there is no background thread available, it was blocking. We found out “s.IoDispatcherForceSynchronousScatter” and setting it to 1 looks to have removed the deadlock. We also saw GIoDispatcherCanDecompressOnStarvation, which is active by default and, from the comment should have prevented it but apparently not.