Data race detected when try to Destroy ChaosCacheManager at Runtime

Development Programming & Scripting C++
,
1

Hello folks,
I needed to dynamicly create and destroy ChaosCacheActor’s in my game. I’ve implemented it and it’s working good. But from time to time I’m getting race detected error crache while destroying ChaosCacheManager at runtime. Looks like it tries to remove delegate instance of ChaosCacheManager when there is already a thread is listening for that instance. Could somebody help please, how could I safely destroy ChaosCacheManager actor? I’m using UE 5.3

This is callstack I’m getting:

Data race detected: acquiring write access on thread 12964 concurrently with 1 writers on thread 9288:
LogOutputDevice: Error: Current thread 12964 callstack:
LogOutputDevice: Error: [Callstack] 0x00007ff9f42a7b11 UnrealEditor-Chaos.dll!FMRSWRecursiveAccessDetector::GetCurrentThreadCallstack() [...\Runtime\Core\Public\Misc\MTAccessDetector.h:373]
LogOutputDevice: Error: [Callstack] 0x00007ff9f4c4204c UnrealEditor-Chaos.dll!`FMRSWRecursiveAccessDetector::AcquireWriteAccess'::`11'::<lambda_6>::operator()() [...\Runtime\Core\Public\Misc\MTAccessDetector.h:595]
LogOutputDevice: Error: [Callstack] 0x00007ff9f42a2909 UnrealEditor-Chaos.dll!TDelegateAccessHandlerBase<FNotThreadSafeDelegateMode>::FWriteAccessScope::FWriteAccessScope() [...\Runtime\Core\Public\Delegates\DelegateAccessHandler.h:139]
LogOutputDevice: Error: [Callstack] 0x00007ff9f469638b UnrealEditor-Chaos.dll!TMulticastDelegateBase<FDefaultDelegateUserPolicy>::RemoveDelegateInstance() [...\Runtime\Core\Public\Delegates\MulticastDelegateBase.h:297]
LogOutputDevice: Error: [Callstack] 0x00007ff9ea83c061 UnrealEditor-ChaosCaching.dll!AChaosCacheManager::EndEvaluate() [...\Plugins\Experimental\ChaosCaching\Source\ChaosCaching\Private\Chaos\CacheManagerActor.cpp:493]
LogOutputDevice: Error: [Callstack] 0x00007ff9ea83c287 UnrealEditor-ChaosCaching.dll!AChaosCacheManager::EndPlay() [...\Plugins\Experimental\ChaosCaching\Source\ChaosCaching\Private\Chaos\CacheManagerActor.cpp:471]
LogOutputDevice: Error: [Callstack] 0x00007ffa0a886498 UnrealEditor-Engine.dll!AActor::Destroyed() [...\Runtime\Engine\Private\Actor.cpp:2794]
LogOutputDevice: Error: [Callstack] 0x00007ffa0b9379d8 UnrealEditor-Engine.dll!UWorld::DestroyActor() [...\Runtime\Engine\Private\LevelActor.cpp:883]
LogOutputDevice: Error: [Callstack] 0x00007ffa0a8863b6 UnrealEditor-Engine.dll!AActor::Destroy() [...\Runtime\Engine\Private\Actor.cpp:4742]
LogOutputDevice: Error: [Callstack] 0x00007ff9bd1efbbf UnrealEditor-MyGame.dll!AMyGameMode::ClearYellowDogActorsExisted() [...\MyGame\Source\MyGame\MyGameGameMode.cpp:93]
LogOutputDevice: Error: [Callstack] 0x00007ffa794bd890 UnrealEditor-CoreUObject.dll!UFunction::Invoke() [...\Runtime\CoreUObject\Private\UObject\Class.cpp:6665]
LogOutputDevice: Error: [Callstack] 0x00007ffa797aca0c UnrealEditor-CoreUObject.dll!UObject::CallFunction() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1139]
LogOutputDevice: Error: [Callstack] 0x00007ffa797c6964 UnrealEditor-CoreUObject.dll!UObject::ProcessContextOpcode() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:3094]
LogOutputDevice: Error: [Callstack] 0x00007ffa797cac48 UnrealEditor-CoreUObject.dll!ProcessLocalScriptFunction() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1209]
LogOutputDevice: Error: [Callstack] 0x00007ffa7979a291 UnrealEditor-CoreUObject.dll!ProcessScriptFunction<void (__cdecl*)(UObject *,FFrame &,void *)>() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1039]
LogOutputDevice: Error: [Callstack] 0x00007ffa797ca671 UnrealEditor-CoreUObject.dll!ProcessLocalFunction() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1279]
LogOutputDevice: Error: [Callstack] 0x00007ffa797cac48 UnrealEditor-CoreUObject.dll!ProcessLocalScriptFunction() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1209]
LogOutputDevice: Error: [Callstack] 0x00007ffa7979a291 UnrealEditor-CoreUObject.dll!ProcessScriptFunction<void (__cdecl*)(UObject *,FFrame &,void *)>() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1039]
LogOutputDevice: Error: [Callstack] 0x00007ffa797ca671 UnrealEditor-CoreUObject.dll!ProcessLocalFunction() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1279]
LogOutputDevice: Error: [Callstack] 0x00007ffa797cac48 UnrealEditor-CoreUObject.dll!ProcessLocalScriptFunction() [...\Sourc
LogOutputDevice: Error: Stack: 
LogOutputDevice: Error: [Callstack] 0x00007ff9f42a2909 UnrealEditor-Chaos.dll!TDelegateAccessHandlerBase<FNotThreadSafeDelegateMode>::FWriteAccessScope::FWriteAccessScope() [...\Runtime\Core\Public\Delegates\DelegateAccessHandler.h:139]
LogOutputDevice: Error: [Callstack] 0x00007ff9f469638b UnrealEditor-Chaos.dll!TMulticastDelegateBase<FDefaultDelegateUserPolicy>::RemoveDelegateInstance() [...\Runtime\Core\Public\Delegates\MulticastDelegateBase.h:297]
LogOutputDevice: Error: [Callstack] 0x00007ff9ea83c061 UnrealEditor-ChaosCaching.dll!AChaosCacheManager::EndEvaluate() [...\Plugins\Experimental\ChaosCaching\Source\ChaosCaching\Private\Chaos\CacheManagerActor.cpp:493]
LogOutputDevice: Error: [Callstack] 0x00007ff9ea83c287 UnrealEditor-ChaosCaching.dll!AChaosCacheManager::EndPlay() [...\Plugins\Experimental\ChaosCaching\Source\ChaosCaching\Private\Chaos\CacheManagerActor.cpp:471]
LogOutputDevice: Error: [Callstack] 0x00007ffa0a886498 UnrealEditor-Engine.dll!AActor::Destroyed() [...\Runtime\Engine\Private\Actor.cpp:2794]
LogOutputDevice: Error: [Callstack] 0x00007ffa0b9379d8 UnrealEditor-Engine.dll!UWorld::DestroyActor() [...\Runtime\Engine\Private\LevelActor.cpp:883]
LogOutputDevice: Error: [Callstack] 0x00007ffa0a8863b6 UnrealEditor-Engine.dll!AActor::Destroy() [...\Runtime\Engine\Private\Actor.cpp:4742]
LogOutputDevice: Error: [Callstack] 0x00007ff9bd1efbbf UnrealEditor-MyGame.dll!AMyGameGameMode::ClearYellowDogActorsExisted() [...\MyGame\Source\MyGame\MyGameGameMode.cpp:93]
LogOutputDevice: Error: [Callstack] 0x00007ffa794bd890 UnrealEditor-CoreUObject.dll!UFunction::Invoke() [...\Runtime\CoreUObject\Private\UObject\Class.cpp:6665]
LogOutputDevice: Error: [Callstack] 0x00007ffa797aca0c UnrealEditor-CoreUObject.dll!UObject::CallFunction() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1139]
LogOutputDevice: Error: [Callstack] 0x00007ffa797c6964 UnrealEditor-CoreUObject.dll!UObject::ProcessContextOpcode() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:3094]
LogOutputDevice: Error: [Callstack] 0x00007ffa797cac48 UnrealEditor-CoreUObject.dll!ProcessLocalScriptFunction() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1209]
LogOutputDevice: Error: [Callstack] 0x00007ffa7979a291 UnrealEditor-CoreUObject.dll!ProcessScriptFunction<void (__cdecl*)(UObject *,FFrame &,void *)>() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1039]
LogOutputDevice: Error: [Callstack] 0x00007ffa797ca671 UnrealEditor-CoreUObject.dll!ProcessLocalFunction() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1279]
LogOutputDevice: Error: [Callstack] 0x00007ffa797cac48 UnrealEditor-CoreUObject.dll!ProcessLocalScriptFunction() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1209]
LogOutputDevice: Error: [Callstack] 0x00007ffa7979a291 UnrealEditor-CoreUObject.dll!ProcessScriptFunction<void (__cdecl*)(UObject *,FFrame &,void *)>() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1039]
LogOutputDevice: Error: [Callstack] 0x00007ffa797ca671 UnrealEditor-CoreUObject.dll!ProcessLocalFunction() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1279]
LogOutputDevice: Error: [Callstack] 0x00007ffa797cac48 UnrealEditor-CoreUObject.dll!ProcessLocalScriptFunction() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1209]
LogOutputDevice: Error: [Callstack] 0x00007ffa797c9c07 UnrealEditor-CoreUObject.dll!UObject::ProcessInternal() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1306]
LogOutputDevice: Error: [Callstack] 0x00007ffa794bd890 UnrealEditor-CoreUObject.dll!UFunction::Invoke() [...\Runtime\CoreUObject\Private\UObject\Class.cpp:6665]
LogOutputDevice: Error: [Callstack] 0x00007ffa797c871f UnrealEditor-CoreUObject.dll!UObject::ProcessEvent() [...\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:2145]
LogOutputDevice: Error: [Callstack] 0x00007ffa0a8abd6d UnrealEditor-Engine.dll!AActor::ProcessEvent() [...\Runtime\Engine\Private\Actor.cpp:1122]
LogOutputDevice: Error: [Callstack] 0x00007ffa0a8adc20 UnrealEditor-Engine.dll!AActor::ReceiveHit() [...\Intermediate\Build\Win64\UnrealEditor\Inc\Engine\UHT\Actor.gen.cpp:2498]
LogOutputDevice: Error: [Callstack] 0x00007ffa0a8a5be2 UnrealEditor-Engine.dll!AActor::NotifyHit() [...\Runtime\Engine\Private\Actor.cpp:2060]
LogOutputDevice: Error: [Callstack] 0x00007ffa0a8a0fd1 UnrealEditor-Engine.dll!AActor::InternalDispatchBlockingHit() [...\Runtime\Engine\Private\Actor.cpp:3021]
LogOutputDevice: Error: [Callstack] 0x00007ffa0a8872b4 UnrealEditor-Engine.dll!AActor::DispatchBlockingHit() [...\Runtime\Engine\Private\Actor.cpp:3037]
LogOutputDevice: Error: [Callstack] 0x00007ffa0b16b2f2 UnrealEditor-Engine.dll!UPrimitiveComponent::DispatchBlockingHit() [...\Runtime\Engine\Private\Components\PrimitiveComponent.cpp:2870]
LogOutputDevice: Error: [Callstack] 0x00007ffa0b19a4fc UnrealEditor-Engine.dll!UPrimitiveComponent::MoveComponentImpl() [...\Runtime\Engine\Private\Components\PrimitiveComponent.cpp:2834]
LogOutputDevice: Error: [Callstack] 0x00007ffa0b19ae1c UnrealEditor-Engine.dll!UMovementComponent::MoveUpdatedComponentImpl() [...\Runtime\Engine\Private\Components\MovementComponent.cpp:531]
LogOutputDevice: Error: [Callstack] 0x00007ffa0b1bd966 UnrealEditor-Engine.dll!UProjectileMovementComponent::TickComponent() [...\Runtime\Engine\Private\Components\ProjectileMovementComponent.cpp:233]
LogOutputDevice: Error: [Callstack] 0x00007ffa0af9f428 UnrealEditor-Engine.dll!FActorComponentTickFunction::ExecuteTickHelper<`FActorComponentTickFunction::ExecuteTick'::`2'::<lambda_1> >() [...\Runtime\Engine\Classes\GameFramework\Actor.h:4490]
LogOutputDevice: Error: [Callstack] 0x00007ffa0b0ae255 UnrealEditor-Engine.dll!FActorComponentTickFunction::ExecuteTick() [...\Runtime\Engine\Private\Components\ActorComponent.cpp:1088]
LogOutputDevice: Error: [Callstack] 0x00007ffa0c88b82b UnrealEditor-Engine.dll!FTickFunctionTask::DoTask() [...\Runtime\Engine\Private\TickTaskManager.cpp:278]
LogOutputDevice: Error: [Callstack] 0x00007ffa0c892d72 UnrealEditor-Engine.dll!TGraphTask<FTickFunctionTask>::ExecuteTask() [...\Runtime\Core\Public\Async\TaskGraphInterfaces.h:1265]
LogOutputDevice: Error: [Callstack] 0x00007ffa14ef25af UnrealEditor-Core.dll!FNamedTaskThread::ProcessTasksNamedThread() [...\Runtime\Core\Private\Async\TaskGraph.cpp:758]
LogOutputDevice: Error: [Callstack] 0x00007ffa14ef29cb UnrealEditor-Core.dll!FNamedTaskThread::ProcessTasksUntilIdle() [...\Runtime\Core\Private\Async\TaskGraph.cpp:660]
LogOutputDevice: Error: [Callstack] 0x00007ffa0c8ba2b9 UnrealEditor-Engine.dll!FTickTaskSequencer::ReleaseTickGroup() [...\Runtime\Engine\Private\TickTaskManager.cpp:573]
LogOutputDevice: Error: [Callstack] 0x00007ffa0c8c3c4e UnrealEditor-Engine.dll!FTickTaskManager::RunTickGroup() [...\Runtime\Engine\Private\TickTaskManager.cpp:1583]
LogOutputDevice: Error: [Callstack] 0x00007ffa0b97cd39 UnrealEditor-Engine.dll!UWorld::RunTickGroup() [...\Runtime\Engine\Private\LevelTick.cpp:771]
LogOutputDevice: Error: [Callstack] 0x00007ffa0b98a6d7 UnrealEditor-Engine.dll!UWorld::Tick() [...\Runtime\Engine\Private\LevelTick.cpp:1530]
LogOutputDevice: Error: [Callstack] 0x00007ffa0f355658 UnrealEditor-UnrealEd.dll!UEditorEngine::Tick() [...\Editor\UnrealEd\Private\EditorEngine.cpp:1924]
LogOutputDevice: Error: [Callstack] 0x00007ffa0ffad8f6 UnrealEditor-UnrealEd.dll!UUnrealEdEngine::Tick() [...\Editor\UnrealEd\Private\UnrealEdEngine.cpp:531]
LogOutputDevice: Error: [Callstack] 0x00007ff7aa9c8f52 UnrealEditor.exe!FEngineLoop::Tick() [...\Runtime\Launch\Private\LaunchEngineLoop.cpp:5825]
LogOutputDevice: Error: [Callstack] 0x00007ff7aa9ed33c UnrealEditor.exe!GuardedMain() [...\Runtime\Launch\Private\Launch.cpp:188]
LogOutputDevice: Error: [Callstack] 0x00007ff7aa9ed42a UnrealEditor.exe!GuardedMainWrapper() [...\Runtime\Launch\Private\Windows\LaunchWindows.cpp:118]
LogOutputDevice: Error: [Callstack] 0x00007ff7aa9f0904 UnrealEditor.exe!LaunchWindowsStartup() [...\Runtime\Launch\Private\Windows\LaunchWindows.cpp:258]
LogOutputDevice: Error: [Callstack] 0x00007ff7aaa06a44 UnrealEditor.exe!WinMain() [...\Runtime\Launch\Private\Windows\LaunchWindows.cpp:298]
LogOutputDevice: Error: [Callstack] 0x00007ff7aaa09e3a UnrealEditor.exe!__scrt_common_main_seh() [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288]
LogOutputDevice: Error: [Callstack] 0x00007ffabd477374 KERNEL32.DLL!UnknownFunction []
LogOutputDevice: Error: [Callstack] 0x00007ffabf47cc91 ntdll.dll!UnknownFunction []
2

Our team is seeing similar AcquireWriteAccess issues in various places, such as at startup for some OVR delegates. It’s leading me to believe that the issue is with the access detector class itself, rather than an actual multithreading bug.

In the example below, I verified that all access to the UOvrPlatformSubsystem happens in the game thread, yet I’m still hitting the ensure.

Ensure condition failed: ReaderIndex != INDEX_NONE [File:D:/a2/main/Engine/Source/Runtime/Core/Public/Misc/MTAccessDetector.h] [Line: 570]
Either a race detected (1 reader(s) on another thread(s) while acquiring write access on the current thread) or the access detector was trivially relocated:
Current thread 11137 callstack:
libUnreal.so(0x000000000D9BCC10)!FMRSWRecursiveAccessDetector::AcquireWriteAccess()::{lambda()#1}::operator()() const
libUnreal.so(0x000000000D9BCA28)!TDelegateAccessHandlerBase::FWriteAccessScope::FWriteAccessScope(FMRSWRecursiveAccessDetector&)
libUnreal.so(0x00000000164B65CC)!UOvrPlatformSubsystem::RemoveRequestDelegate(unsigned long)
libUnreal.so(0x00000000164BC310)!UOvrPlatformSubsystem::OnReceiveMessage(ovrMessage*)
libUnreal.so(0x00000000164BC430)!non-virtual thunk to UOvrPlatformSubsystem::Tick(float)
libUnreal.so(0x0000000014A83ECC)!FTickableGameObject::TickObjects(UWorld*, int, bool, float)
libUnreal.so(0x0000000013C6DFE8)!UGameEngine::Tick(float, bool)
libUnreal.so(0x000000001560C024)!FEngineLoop::Tick()
libUnreal.so(0x000000001560391C)!AndroidMain(android_app*)
libUnreal.so(0x0000000015613DFC)!android_main()
libUnreal.so(0x000000001563B150)Unknown
libc.so(0x00000000000DF50C)Unknown
libc.so(0x000000000008B570)Unknown

3

Hi DanH_AA, thanks for your response.
Looks like I’ve managed to figure out how to solve that issue. I am currently in progress fixing it. But this issue is completely related to my custom changes with Chaos Destruction system for UE 5.3.

Simply speaking during spawning and destruction of ChaosCachePlayer classes, callbacks are added and removed respectively for 4 different delegates. These are places for this

CacheManagerActor::BeginEvaluate => lines 360-363 
CacheManagerActor::EndEvaluate => lines 492-495

Delegates are from PBDRigidsSolver(more exactly are from its parent FPhysicsSolverEvents)

FSolverPreAdvance EventPreSolve;
FSolverPreBuffer EventPreBuffer;
FSolverPostAdvance EventPostSolve;
FSolverTeardown EventTeardown;

During Adding/Removing corresponding callbacks to those delagates, may come a situation when there is already another background/fpreground worker threads running these delegates(thread called Broadcast). There are not so many places where those delegates are used. In my case everything is inside of PBDRigidsSolver.
So, if you synchronize access to those delegates, the error will disappear. And I didn’t notice yet any drawbacks of this.

I’ve tested it with Epic code, it’s worked out. But There are a lot of changes to be done to fix it. At least custom Physics solver has to be created. I am doing it right now.

As per my understanding, it is forbidden to add/remove new delegate instances if there are threads reading already existed instances.

Hope it brings some clarity for this

1 Like