Sporadic Build error with ZenLoader when using MultiProcessCook

Development Pipeline & Plugins
, ,
1

Hello. We are running into some unique build errors that give us this exception

LogOutputDevice: Error: [CookWorker 0]: Ensure condition failed: (GetLoaderType() != ELoaderType::ZenLoader) || !HasAnyFlags(RF_NeedLoad)

The weird issue is that it seems to happen at random for both Linux and Win64 builds, with no real direction on where to look other than MultiProcessCook failed. We’ve had 3 builds fail in a row, as well as many more succeed without the issue, not changing anything.

Error trying to run Commandlet BuildCookRunWithAutoMultiProcessCook. Still attempting to copy additional logs to output directory. Error: Cook fail

Any idea what may be the issue or where to start looking to fix this? It doesn’t seem to consistently error on any specific asset it is cooking either, so unsure if that is specifically the culprit.

[Attachment Removed]

2

Steps to Reproduce
Have MultiProcessCook enabled, build for Linux or Win64. Sometimes when cooking assets, fails with error

LogOutputDevice: Error: [CookWorker 0]:

Script Stack (0 frames) :

LogOutputDevice: Error: [CookWorker 0]: Ensure condition failed: (GetLoaderType() != ELoaderType::ZenLoader) || !HasAnyFlags(RF_NeedLoad)  [File:C:\fbroot\Engine\Source\Runtime\CoreUObject\Private\UObject\Obj.cpp] [Line: 1314]

Never consistent on which asset this fails on.

[Attachment Removed]

3

Hi Cooper,

This happens when an object somehow gets to the postload without the object having been deserialized first.

Since this is not consistent, this is clearly a timing problem. There is a few place where blueprint code puts back RF_NeedLoad for various reasons so it might be one of those place causing the issue. It could also be GC related.

Can you get a callstack of when this happens?

Does it happen on the asset types consistently?

As there been a GC near the place where this happens?

Can you print the object class and object name when this happens? We might be able to identify a pattern that way.

Thanks

Danny

[Attachment Removed]

4

Hi Danny,

It does seem to happen on a child UPrimaryDataAsset type we set up consistently. One of the examples being this:

LogCook: Display: [CookWorker 0]: Cooking /Enemy_Enemy_Boss_Harvester/Gameplay/ActionSets/ActSet_Boomerang/Boomerang/ChronoDef_Harvester_Boomerang
LogOutputDevice: Error: [CookWorker 0]:
Script Stack (0 frames) :
LogOutputDevice: Error: [CookWorker 0]: Ensure condition failed: (GetLoaderType() != ELoaderType::ZenLoader) || !HasAnyFlags(RF_NeedLoad)  [File:C:\fbroot\Engine\Source\Runtime\CoreUObject\Private\UObject\Obj.cpp] [Line: 1314]

I do see GC happening periodically throughout the log, though it doesn’t seem to have the error trigger anytime near after or before it is called, being able to cook many different assets before it gets the error.

[Attachment Removed]

5

My next step would be trying to track down RF_NeedLoad object flag modification on objects of these types and log them with a callstack stack with additional code in both AtomicallySetFlags and AtomicallyClearFlags. Also making sure to output the full object path or at least the pointer address to make sure you can correlate the PostLoad error with the RF_NeedLoad changes once you hit one.

It might reveal the culprit.

You can look up FPlatformStackWalk::CaptureStackBackTrace and FPlatformStackWalk::ProgramCounterToHumanReadableString. We do this at a few places in the engine to output callstacks.

Hope this helps.

[Attachment Removed]