Rare occurence of check(!HasAnyClassFlags(CLASS_TokenStreamAssembled)) in game

Development Platform & Builds
, ,
1

Hello!

We have a few occurences of !HasAnyClassFlags(CLASS_TokenStreamAssembled).

It happen on PC & PlayStation 5 and it’s not always the same BP. We enabled data-breakpoint based on name but unfortunately we couldn’t get hit to reproduce.

I was wondering if you had any idea on how we could debug this issue, or if that’s something already on your radar.

Thank you.

[Attachment Removed]

2

Steps to Reproduce[Attachment Removed]

3

Hey there, I’ve had a chat with a colleague, and we’d like to collect some more info to see if the crash is caused by the GameThread and async loading thread either…

  1. calling AssembleReferenceTokenStream on two related classes (parent and derived). It’s not a concrete theory yet because there is a mutex in place that should prevent multiple BPGCs from assembling their reference token streams simultaneously.
  2. or, the GameThread executing BPGC::PostLoad like in your callstack, while the async loading thread is serializing the BPGC. CL 24279266, which is still in UE 5.6 should prevent that. That has since been replaced by CL 45909679 in UE 5.7.

So far, those are two ideas we have. I have a few questions.

Q1: Judging from the call stack you provided, it crashed on the AssembleReferenceTokenStreamInternal call for the BlueprintGeneratedClass itself and not one of its super-classes. Can you confirm that was the case? And the following check is the one that fails?

checkf(!HasAnyClassFlags(CLASS_TokenStreamAssembled), TEXT("GC schema already assembled for class '%s'"), *GetPathName()); // recursion here is probably badQ2: Another thing I want to rule out is whether you have any engine modifications surrounding async loading. Especially in AsyncLoading2.cpp. If you have any, it would be good to be aware, since the file is changed frequently.

Q3: Another unlikely theory I want to rule out: have you overwritten UObject::IsPostLoadThreadSafe() on any class in your project? If so, is it related to the objects that trigger the crash?

Q4: Since checks are compiled out of shipping builds, is this impacting a live game or are you just running into this during development?

[Attachment Removed]

4

Thanks for those answers and for providing the async loading thread’s callstack!

Knowing that both threads are PostLoading a BPGC is useful. We’d like to figure out if they are loading the same BPGC or not. If they’re different BPGC’s it’s useful to find out whether they’re related (same native parent class, for example).

Since the crash it’s hard to repro, can you put some debug code in place for the next time the crash happens? This will be hacky and I recommend cleaning it up at some point. Store the class name being processed in AssembleReferenceTokenStream, based on current thread:

// START engine modification
FString GAssemblingReferenceTokenStreamForClass_GameThread = "";
FString GAssemblingReferenceTokenStreamForClass_AsyncThread = "";
// END engine modification
void UClass::AssembleReferenceTokenStream(bool bForce)
{
	// START engine modification
#if !UE_BUILD_SHIPPING
	FString& AssemblingClassPerThread = IsInGameThread() ? GAssemblingReferenceTokenStreamForClass_GameThread : GAssemblingReferenceTokenStreamForClass_AsyncThread;
	TGuardValue<FString> ScopedClassNameSet(AssemblingClassPerThread, GetName());
#endif
	// END engine modification
	...
}

And modify the check in AssembleReferenceTokenStreamInternal:

// START engine mod
checkf(!HasAnyClassFlags(CLASS_TokenStreamAssembled), TEXT("GC schema already assembled for class '%s'. GameThread assembling: %s. AsyncThread assembling: %s"), *GetPathName(), *GAssemblingReferenceTokenStreamForClass_GameThread, *GAssemblingReferenceTokenStreamForClass_AsyncThread); // recursion here is probably bad
// END engine mod

When the crash happens again, and both the GT and AsyncThread are inside AssembleReferenceTokenStream(), and hopefully from two BPGCs like in your callstacks so far, we’ll learn which two BPGCs. That’ll help us hone in on the problem.

[Attachment Removed]

5

Is it possible to edit your UObject::ConditionalPostLoad and add some validation code, see if something get caught.

if (IsInActualLoadingThread())
{
   // make sure we're not trying to postload something that the loader doesn't own
   check(HasAnyInternalFlags(EInternalObjectFlags::AsyncLoadingPhase1);
}
else
{
   // make sure if we're not in the loader, we're not trying to postload something owned by the loader
   check(!HasAnyInternalFlags(EInternalObjectFlags::AsyncLoadingPhase1);
}

Thanks

Danny

[Attachment Removed]

6

Hello!

Q1: Yes this is the line that trigger the check

Q2: We do have very minor changes in that file (mostly editor related) or to had profile tags

Q3: We don’t have any override of IsPostLoadThreadSafe()

Q4: That really hard to tell has I’m not even sure how that would manifest.

Here’s the callstack of the loading thread at the time of crash

... [begining of the stack trimmed as platform specific] ...
[Inline Function] FScopeLock::FScopeLock()
[Inline Function] FScopeLock::FScopeLock()
UClass::AssembleReferenceTokenStream(bool)
UBlueprintGeneratedClass::PostLoad()
UObject::ConditionalPostLoad()
UObject::PostLoadSubobjects(FObjectInstancingGraph*)
UObject::ConditionalPostLoadSubobjects(FObjectInstancingGraph*)
FAsyncPackage2::EventDrivenCreateExport(const FAsyncPackageHeaderData&, int32)
FAsyncPackage2::Event_ProcessExportBundle(FAsyncLoadingThreadState2&, FAsyncPackage2*, int32)
FEventLoadNode2::Execute(FAsyncLoadingThreadState2&)
FAsyncLoadEventQueue2::ExecuteSyncLoadEvents(FAsyncLoadingThreadState2&)
FAsyncLoadingThread2::Run()

Importunely I don’t have the memory of the UClass pointer in the loading thread in our crash dump, so I can’t provide more information

[Attachment Removed]

7

Sure, that’s a good idea.

I’ll keep you posted whenever the crash happens again.

Thank you.

[Attachment Removed]

8

Great! I’ll await your update when it happens again.

[Attachment Removed]