Offline Asset Processing
After upgrading to 5.4.3 (from 5.3.X) we started experiencing hangs when running the DerivedDataCache commandlet on all of our assets on build machines. Happens frequently, but not always.
Specifically, hanging on Skinned assets, and in the two instances I checked out, it was the same although there is nothing special about it. Building it locally in isolation does not hang. Its 160 mb, a metahuman head. We have about 165 skeletal meshes to process. Of those, ~35 are comparably sized metahuman heads. To be clear, there appeared to only be 1 left to process in both cases I looked at (1 was a dump, and the other I remotely attached to the VM processing the assets).
Stack looks like this:
GameEditor-Core.dll!FWindowsPlatformProcess::Sleep(float Seconds) Line 1568 C++
GameEditor-Engine.dll!AsyncCompilationHelpers::FinishCompilation(TFunctionRef<AsyncCompilationHelpers::ICompilable & __cdecl(int)> Getter, int Num, const FText & AssetType, const FLogCategoryBase & LogCategory, TFunctionRef<void __cdecl(AsyncCompilationHelpers::ICompilable *)> PostCompileSingle) Line 212 C++
GameEditor-Engine.dll!FSkinnedAssetCompilingManager::FinishCompilation(TArrayView<USkinnedAsset * const,int> InSkinnedAssets) Line 336 C++
GameEditor-Engine.dll!FSkinnedAssetCompilingManager::FinishAllCompilation() Line 378 C++
[Inline Frame] GameEditor-Engine.dll!Invoke(void(FSkinnedAssetCompilingManager::)()) Line 66 C++
[Inline Frame] GameEditor-Engine.dll!UE::Core::Private::Tuple::TTupleBase<TIntegerSequence>::ApplyAfter(void(FSkinnedAssetCompilingManager::)() &) Line 309 C++
GameEditor-Engine.dll!TBaseRawMethodDelegateInstance<0,FSkinnedAssetCompilingManager,void cdecl(void),FDefaultDelegateUserPolicy>::ExecuteIfSafe() Line 534 C++
[Inline Frame] GameEditor-CoreUObject.dll!TMulticastDelegateBase::Broadcast() Line 254 C++
GameEditor-CoreUObject.dll!TMulticastDelegate<void cdecl(void),FDefaultDelegateUserPolicy>::Broadcast() Line 956 C++
GameEditor-CoreUObject.dll!UE::GC::PreCollectGarbageImpl<1>(EObjectFlags KeepFlags) Line 5412 C++
GameEditor-CoreUObject.dll!UE::GC::FReachabilityAnalysisState::PerformReachabilityAnalysisAndConditionallyPurgeGarbage(bool bReachabilityUsingTimeLimit) Line 5664 C++
[Inline Frame] GameEditor-CoreUObject.dll!UE::GC::CollectGarbageInternal(EObjectFlags) Line 5342 C++
GameEditor-CoreUObject.dll!CollectGarbage(EObjectFlags KeepFlags, bool bPerformFullPurge) Line 6007 C++
GameEditor-UnrealEd.dll!UDerivedDataCacheCommandlet::Main(const FString & Params) Line 983 C++
GameEditor-Cmd.exe!FEngineLoop::PreInitPostStartupScreen(const wchar_t * CmdLine) Line 4228 C++
[Inline Frame] GameEditor-Cmd.exe!FEngineLoop::PreInit(const wchar_t *) Line 4542 C++
[Inline Frame] GameEditor-Cmd.exe!EnginePreInit(const wchar_t *) Line 41 C++
GameEditor-Cmd.exe!GuardedMain(const wchar_t * CmdLine) Line 136 C++
GameEditor-Cmd.exe!GuardedMainWrapper(const wchar_t * CmdLine) Line 146 C++
GameEditor-Cmd.exe!LaunchWindowsStartup(HINSTANCE * hInInstance, HINSTANCE * hPrevInstance, char * formal, int nCmdShow, const wchar_t * CmdLine) Line 290 C++
GameEditor-Cmd.exe!WinMain(HINSTANCE * hInInstance, HINSTANCE__ * hPrevInstance, char * pCmdLine, int nCmdShow) Line 330 C++
[Inline Frame] GameEditor-Cmd.exe!invoke_main() Line 102 C++
We were able to eliminate this issue by disabling AsyncSkinnedAssetCompilation, in the DefaultEditorPerProjectUserSettings config file, like so:
[/Script/UnrealEd.EditorExperimentalSettings]
bEnableAsyncSkinnedAssetCompilation=False
;bEnableAsyncSkinnedAssetCompilation=True
There is a similar question that may be related →
Here the user appeared to resolve the issue by increasing the abailible system resources. I can assure you that is not the issue here; The machine has over 200 gb of free physical memory, and plenty of availible CPU.
We are not blocked, but of course by eliminating AsyncSkinnedAssetCompilation our builds take longer, and we’d love to turn it back on.
Thanks,
Doug