Crash when building morph targets due to discrepancy between StaticFindObject and StaticFindObjectFastInternal

Hey folks!

We just ran into this crash while trying to cook our game content:

Runnable thread Foreground Worker #0 crashed.
begin: stack for UAT
=== Critical error: ===
 
Assertion failed: IsInGameThread() [File:H:\TQ2\Engine\Source\Runtime\CoreUObject\Private\UObject\UObjectArray.cpp] [Line: 384]
 
 
 
[Callstack] 0x00007ff952f3efc8 UnrealEditor-Core.dll!FDebug::CheckVerifyFailedImpl2() [H:\TQ2\Engine\Source\Runtime\Core\Private\Misc\AssertionMacros.cpp:745]
[Callstack] 0x00007ffa0ee9aafd UnrealEditor-CoreUObject.dll!FUObjectArray::FreeUObjectIndex() [H:\TQ2\Engine\Source\Runtime\CoreUObject\Private\UObject\UObjectArray.cpp:384]
[Callstack] 0x00007ffa0eeb7e56 UnrealEditor-CoreUObject.dll!StaticAllocateObject() [H:\TQ2\Engine\Source\Runtime\CoreUObject\Private\UObject\UObjectGlobals.cpp:3841]
[Callstack] 0x00007ffa0eeb90ec UnrealEditor-CoreUObject.dll!StaticConstructObject_Internal() [H:\TQ2\Engine\Source\Runtime\CoreUObject\Private\UObject\UObjectGlobals.cpp:4984]
[Callstack] 0x00007ff97203fa2c UnrealEditor-SkeletalMeshUtilitiesCommon.dll!BuildMorphTargetsInternal() [H:\TQ2\Engine\Source\Developer\SkeletalMeshUtilitiesCommon\Private\LODUtilities.cpp:3359]
[Callstack] 0x00007ff93fbcbada UnrealEditor-MeshBuilder.dll!FinalizeContext() [H:\TQ2\Engine\Source\Developer\MeshBuilder\Private\SkeletalMeshBuilder.cpp:757]
[Callstack] 0x00007ff93fbbd849 UnrealEditor-MeshBuilder.dll!FSkeletalMeshBuilder::Build() [H:\TQ2\Engine\Source\Developer\MeshBuilder\Private\SkeletalMeshBuilder.cpp:903]
[Callstack] 0x00007ff93fbd535e UnrealEditor-MeshBuilder.dll!FMeshBuilderModule::BuildSkeletalMesh() [H:\TQ2\Engine\Source\Developer\MeshBuilder\Private\MeshBuilderModule.cpp:83]
[Callstack] 0x00007ff94f8309fe UnrealEditor-Engine.dll!USkeletalMesh::BuildLODModel() [H:\TQ2\Engine\Source\Runtime\Engine\Private\SkeletalMesh.cpp:6583]
[Callstack] 0x00007ff94f8b1d39 UnrealEditor-Engine.dll!FSkeletalMeshRenderData::Cache() [H:\TQ2\Engine\Source\Runtime\Engine\Private\SkeletalMeshRenderData.cpp:327]
[Callstack] 0x00007ff94f830d0e UnrealEditor-Engine.dll!USkeletalMesh::CacheDerivedData() [H:\TQ2\Engine\Source\Runtime\Engine\Private\SkeletalMesh.cpp:5533]
[Callstack] 0x00007ff94f83fb86 UnrealEditor-Engine.dll!USkeletalMesh::ExecutePostLoadInternal() [H:\TQ2\Engine\Source\Runtime\Engine\Private\SkeletalMesh.cpp:4090]
[Callstack] 0x00007ff9501d452f UnrealEditor-Engine.dll!FSkinnedAssetAsyncBuildWorker::DoWork() [H:\TQ2\Engine\Source\Runtime\Engine\Private\SkinnedAssetAsyncCompileUtils.cpp:41]
[Callstack] 0x00007ff94e8c2a82 UnrealEditor-Engine.dll!FAsyncTaskBase::DoWork() [H:\TQ2\Engine\Source\Runtime\Core\Public\Async\AsyncWork.h:289]
[Callstack] 0x00007ff94e8c24be UnrealEditor-Engine.dll!FAsyncTaskBase::DoThreadedWork() [H:\TQ2\Engine\Source\Runtime\Core\Public\Async\AsyncWork.h:313]
[Callstack] 0x00007ff952ffb2c8 UnrealEditor-Core.dll!FQueuedThreadPoolWrapper::FScheduledWork::DoThreadedWork() [H:\TQ2\Engine\Source\Runtime\Core\Public\Misc\QueuedThreadPoolWrapper.h:110]
[Callstack] 0x00007ff94fe956f5 UnrealEditor-Engine.dll!FMemoryBoundQueuedThreadPoolWrapper::FMemoryBoundScheduledWork::DoThreadedWork() [H:\TQ2\Engine\Source\Runtime\Engine\Private\AssetCompilingManager.cpp:363]
[Callstack] 0x00007ff7cc4af14c UnrealEditor-Cmd.exe!`LowLevelTasks::FTask::Init<`FQueuedLowLevelThreadPool::AddQueuedWork'::`2'::<lambda_1> >'::`13'::<lambda_1>::operator()() [H:\TQ2\Engine\Source\Runtime\Core\Public\Async\Fundamental\Task.h:499]
[Callstack] 0x00007ff7cc4b5885 UnrealEditor-Cmd.exe!LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __cdecl(bool),48>::TTaskDelegateImpl<`LowLevelTasks::FTask::Init<`FQueuedLowLevelThreadPool::AddQueuedWork'::`2'::<lambda_1> >'::`13'::<lambda_1>,0>::CallAndMove() [H:\TQ2\Engine\Source\Runtime\Core\Public\Async\Fundamental\TaskDelegate.h:171]
[Callstack] 0x00007ff952e0cba7 UnrealEditor-Core.dll!LowLevelTasks::FTask::ExecuteTask() [H:\TQ2\Engine\Source\Runtime\Core\Public\Async\Fundamental\Task.h:627]
[Callstack] 0x00007ff952e0c904 UnrealEditor-Core.dll!LowLevelTasks::FScheduler::ExecuteTask() [H:\TQ2\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:397]
[Callstack] 0x00007ff952e1b51e UnrealEditor-Core.dll!LowLevelTasks::FScheduler::WorkerLoop() [H:\TQ2\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:757]
[Callstack] 0x00007ff952e04535 UnrealEditor-Core.dll!`LowLevelTasks::FScheduler::CreateWorker'::`2'::<lambda_1>::operator()() [H:\TQ2\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:220]
[Callstack] 0x00007ff953274ca4 UnrealEditor-Core.dll!FThreadImpl::Run() [H:\TQ2\Engine\Source\Runtime\Core\Private\HAL\Thread.cpp:69]
[Callstack] 0x00007ff9534cad0d UnrealEditor-Core.dll!FRunnableThreadWin::Run() [H:\TQ2\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:159]
[Callstack] 0x00007ff9534cabbf UnrealEditor-Core.dll!FRunnableThreadWin::GuardedRun() [H:\TQ2\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:79]
[Callstack] 0x00007ffa9bcfe8d7 KERNEL32.DLL!UnknownFunction []
 
Crash in runnable thread Foreground Worker #0
end: stack for UAT

It looks like for some reason the NewObject call in BuildMorphTargetsInternal is trying to reuse an existing object, which causes this crash when it happens outside of the game thread.

There is a specific check right before the NewObject call which looks like it should prevent this issue:

if (!IsInGameThread())
{
    //TODO remove this code when overriding a UObject will be allow outside of the game thread
    //We currently need to avoid overriding an existing asset outside of the game thread
    UObject* ExistingMorphTarget = StaticFindObject(UMorphTarget::StaticClass(), BaseSkelMesh, *ShapeName);
    const auto SanityCheck = StaticFindObjectFastInternal(UMorphTarget::StaticClass(), BaseSkelMesh, ObjectName, EFindObjectFlags::ExactClass );
    checkf(ExistingMorphTarget == SanityCheck, TEXT("%p != %p"), ExistingMorphTarget, SanityCheck);
    if (ExistingMorphTarget)
    {
        //make sure the object is not standalone or transactional
        ExistingMorphTarget->ClearFlags(RF_Standalone | RF_Transactional);
        //Move this object in the transient package
        ExistingMorphTarget->Rename(nullptr, GetTransientPackage(), REN_DoNotDirty | REN_DontCreateRedirectors | REN_NonTransactional);
        ExistingMorphTarget = nullptr;
    }
}
 
MorphTarget = NewObject<UMorphTarget>(BaseSkelMesh, ObjectName);

But for some reason, the StaticFindObject call does not find the existing object.

As a sanity check, I added the following code immediately after StaticFindObject:

const auto SanityCheck = StaticFindObjectFastInternal(UMorphTarget::StaticClass(), BaseSkelMesh, ObjectName, EFindObjectFlags::ExactClass );
checkf(ExistingMorphTarget == SanityCheck, TEXT("%p != %p"), ExistingMorphTarget, SanityCheck);

And this assert does trigger. So for some reason, the existing object instance can be found by StaticFindObjectFastInternal, but not by StaticFindObject.

Do you have any idea what could be causing this discrepancy and how to fix it?

Cheers,

Dave

Upon further investigation, it looks like the discrepancy comes from this part in StaticFindObjectFast:

This excludes objects still tagged as AsyncLoadingPhase1/2, which excludes the object in question:

Still not entirely sure how best to solve this, though. We could simply replace the StaticFindObject call with a call to StaticFindObjectFastInternal, but that looks like it might introduce threading issues.

Hey there,

There’s a small change that was made recently that should help. In your mainline p4 connection it would be CL: 49749142 and the github link would be https://github.com/EpicGames/UnrealEngine/commit/8c41d3424139c146b217fe3601a3c783313b3500.

Could you try integrating that change first?

Dustin

Hey Dustin, thanks for the reply!

I took a look at the changelist and it looks exactly like the temporary fix we had already applied locally, so I can confirm that it works.