Crash during copy of CustomPrimitiveDataInternal in InitProperties on Async Loading thread

Development Platform & Builds
, ,
1

Hi,

We have noted a couple of crashes in the game while loading data for an instance of a custom component (deriving from UStaticMeshComponent) on the Async Loading thread,

specifically while copying the CustomPrimitiveDataInternal property from the template object in a blueprint (which has some CustomPrimitiveData).

Following is some findings from the crash dump and some extra instrumentation that was temporarily added:

It seems the Source pointer while copying the FCustomPrimitiveData::Data array is saved as null in some register, while after the crash this is no longer the case (the source array shown to be intact),

my assumption is that some other thread happened to touch the CustomPrimitiveDataInternal.Data array at the same time as we fetched and stored the internal data pointer for it on the Async Loading thread.

It happens that the GT was just recently in the process of going over the template UObject and calling PostLoad on it,

following this I found that UPrimitiveComponent::PostLoad calls ResetCustomPrimitiveData which does indeed copy over CustomPrimitiveData.Data to CustomPrimitiveDataInternal.Data and I assume this is what happened while we started this copy.

Currently we have no repro for this and I’m not sure exactly how to go about fixing this since I’m not sure if this is even an allowed scenario or how we got here, our custom component only calls Super in its PostLoad.

Any help with this would be appreciated!

Best,

Robin Krokfors

[Attachment Removed]

2

Hello!

You have 2 options that should help you figure this out.

The first one is to use the Race Detector feature that is new in 5.7. You must compile with instrumentation turned on and run with -racedetector . There are 3 ways to compile with instrumentation

  • Through the targetRules. Applies to a specific target.
// In Project.Target.cs (or ProjectEditor.Target.cs)
WindowsPlatform.bEnableInstrumentation = true;
  • Compile from the command line with -EnableInstrumentation
  • From the BuildConfiguration.xml
<?xml version="1.0" encoding="utf-8" ?>
<Configuration xmlns="https://www.unrealengine.com/BuildConfiguration">
 
  <BuildConfiguration>
    <bEnableInstrumentation>true</bEnableInstrumentation>
  </BuildConfiguration>
  
</Configuration>

The 2nd option is to use the MTAccessDetector classes. There are a bunch of macros to declare and manipulate the detector objects at the end of MTAccessDetector.h. You can find a lot of examples usage spread through the engine code.

I recommend starting with the race detector.

Regards,

Martin

[Attachment Removed]

3

Did you check if it’s the same instance\memory being touched in both thread? The MTAccessDetector checks for concurrency of code sections but it doesn’t mean the data is the same. You should try with the Race Detector.

[Attachment Removed]

4

Can you check if the object in the Gamethread is a CDO? Likely a BP.

[Attachment Removed]

5

One thing you could try would be to move the call to ResetCustomPrimitiveData from PostLoad to UPrimitiveComponent::Serialize while guarding it with Ar.IsLoading(). That would ensure that the CustomPrimitiveDataInternal array is already initialize when the CDO is being used as a template in the ALT .

Now, you might ask about the rest of the code in PostLoad. Most of that code takes care of converting from older version of the class. Based on my research, the code in PostLoad already sets the proper values when loading the data in the cook commandlet so no conversion will ever happen in the runtime. That is assuming that you are using unversioned cook and don’t expect to be able to load old cooked data\paks.

[Attachment Removed]

6

Hi,

Thanks for confirming this resolved the problem. We decided to push this as a fix to prevent future occurrence of the problem. CL52911236

Regards,

Martin

[Attachment Removed]

7

Hi Martin!

Thanks for the tips, I went ahead with the MTAccessDetector in the end since I already had a suspicion on the location of the access.

I temporarily added a new AccessDetector in UObject and took a write access scope in UPrimitiveComponent::ResetCustomPrimitiveData for this,

I also added a read scope in FObjectInitializer::InitProperties on DefaultData since later in the FProperty copy loop it will copy over CustomPrimitiveDataInternal from DefaultData.

I then managed to get this between the GT and ALT for a normal static mesh component with some CustomPrimitiveData:

GT:

Game.exe!FDebug::CheckVerifyFailedImpl2V(const char * Expr=0x00007ff76084e670, const char * File=0x00007ff7607bba50, int Line=0x00000072, const wchar_t * Format=0x00007ff76084e750, char * Args=0x000000da1377bd10)
Game.exe!FDebug::CheckVerifyFailedImpl2(const char * Expr, const char * File, int Line, const wchar_t * Format=0x00007ff76084e750, ...)
[Inline Frame] Game.exe!FRWAccessDetector::AcquireWriteAccess()
[Inline Frame] Game.exe!TScopedWriterDetector<FRWAccessDetector>::{ctor}(FRWAccessDetector &)
[Inline Frame] Game.exe!MakeScopedWriterAccessDetector(FRWAccessDetector &)
Game.exe!UPrimitiveComponent::ResetCustomPrimitiveData()
Game.exe!UPrimitiveComponent::PostLoad()
Game.exe!UStaticMeshComponent::PostLoad()
Game.exe!UObject::ConditionalPostLoad()
[Inline Frame] Game.exe!FAsyncLoadingTickScope2::{ctor}(FAsyncLoadingThread2 & InAsyncLoadingThread)
Game.exe!FAsyncPackage2::Event_DeferredPostLoadExportBundle(FAsyncLoadingThreadState2 & ThreadState={...}, FAsyncPackage2 * Package=0x00000204d2526fa0, int InExportBundleIndex)
Game.exe!FEventLoadNode2::Execute(FAsyncLoadingThreadState2 & ThreadState={...})
[Inline Frame] Game.exe!TGuardValue<enum EInternalObjectFlags,enum EInternalObjectFlags>::{dtor}()
Game.exe!FAsyncLoadEventQueue2::PopAndExecute(FAsyncLoadingThreadState2 & ThreadState={...})
Game.exe!FAsyncLoadingThread2::ProcessLoadedPackagesFromGameThread(FAsyncLoadingThreadState2 & ThreadState={...}, bool & bDidSomething=true, TArrayView<int const ,int> FlushRequestIDs={...})
Game.exe!FAsyncLoadingThread2::TickAsyncLoadingFromGameThread(FAsyncLoadingThreadState2 & ThreadState={...}, bool bUseTimeLimit=true, bool bUseFullTimeLimit=false, double TimeLimit=0.0050000003539025784, TArrayView<int const ,int> FlushRequestIDs={...}, bool & bDidSomething=true)
Game.exe!FAsyncLoadingThread2::ProcessLoadingFromGameThread(FAsyncLoadingThreadState2 & ThreadState={...}, bool bUseTimeLimit=true, bool bUseFullTimeLimit=false, double TimeLimit=0.0050000003539025784)
Game.exe!FAsyncLoadingThread2::ProcessLoading(bool bUseTimeLimit=true, bool bUseFullTimeLimit=false, double TimeLimit)
Game.exe!ProcessAsyncLoading(bool bUseTimeLimit=true, bool bUseFullTimeLimit=false, double TimeLimit)
Game.exe!StaticTick(float DeltaTime=0.00000000, bool bUseFullTimeLimit=false, float AsyncLoadingTime)
Game.exe!UGameEngine::Tick(float DeltaSeconds, bool bIdleMode=false)
Game.exe!FEngineLoop::Tick()
[Inline Frame] Game.exe!EngineTick()
Game.exe!GuardedMain(const wchar_t * CmdLine=0x00000201bb312c80)
Game.exe!LaunchWindowsStartup(HINSTANCE__ * hInInstance=0x000000000000000a, HINSTANCE__ * hPrevInstance=0x0000000000000000, char * __formal=0x0000000000000000, int nCmdShow=0x0ada70b0, const wchar_t * CmdLine=0x00000201bb312c80)
Game.exe!WinMain(HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, char * pCmdLine, int nCmdShow)
[Inline Frame] Game.exe!invoke_main()
Game.exe!__scrt_common_main_seh()
kernel32.dll!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()

ALT:

[Inline Frame] Game.exe!FRWAccessDetector::ReleaseReadAccess()
[Inline Frame] Game.exe!FObjectInitializer::InitProperties::__l2::<lambda_1>::operator()()
[Inline Frame] Game.exe!ScopeExitSupport::TScopeGuard<`FObjectInitializer::InitProperties'::`2'::<lambda_1>>::{dtor}()
Game.exe!FObjectInitializer::InitProperties(UObject * Obj=0x000002053d1e6280, UClass * DefaultsClass=0x00007ff48f88c8a8, UObject * DefaultData=0x00000204a00d41b0, bool bCopyTransientsFromClassDefaults=0x73)
Game.exe!FObjectInitializer::PostConstructInit()
Game.exe!FObjectInitializer::~FObjectInitializer()
Game.exe!StaticConstructObject_Internal(const FStaticConstructObjectParameters & Params)
Game.exe!FAsyncPackage2::EventDrivenCreateExport(const FAsyncPackageHeaderData & Header, int LocalExportIndex)
Game.exe!FAsyncPackage2::Event_ProcessExportBundle(FAsyncLoadingThreadState2 & ThreadState={...}, FAsyncPackage2 * Package=0x00000205480d6280, int InExportBundleIndex=0x00000000)
Game.exe!FEventLoadNode2::Execute(FAsyncLoadingThreadState2 & ThreadState={...})
[Inline Frame] Game.exe!TGuardValue<enum EInternalObjectFlags,enum EInternalObjectFlags>::{dtor}()
Game.exe!FAsyncLoadEventQueue2::PopAndExecute(FAsyncLoadingThreadState2 & ThreadState={...})
Game.exe!FAsyncLoadingThread2::Run()
Game.exe!FRunnableThreadWin::Run()
Game.exe!FRunnableThreadWin::GuardedRun()
kernel32.dll!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()

This seems to indicate that there could be an issue for any PrimitiveComponent if the template object is being loaded at the same time and has some CustomPrimitiveData.

Best,

Robin

[Attachment Removed]

8

Indeed that was checked, in this case both the DefaultData on the ALT and the UStaticMeshComponent on the GT was 0x00000204a00d41b0. Since the AccessDetector object I was using is a new one I had just added to the base UObject class so I expect a unique one per UObject instance.

[Attachment Removed]

9

The component on the GT is indeed from a blueprint (the object used as a Template on the ALT I assume when loading an instance of it)

[Attachment Removed]

10

Thanks for this suggestion I do believe it solved the potential crash scenario,

I have seen no issues with the data either which is as expected since now we are just copying right after it is deserialized.

I am currently not so worried about most other modifications in PostLoad since we don’t expect to load old cooked data outside editor/cook.

So for now this seems to have solved our issue, thanks for the help!

Best,

Robin

[Attachment Removed]