Zen Incremental Cook has almost as long a runtime as a clean cook

anonymous-edc · September 15, 2025, 10:36pm

I’ve been experimenting with the Incremental cook in 5.6

Running a clean cook, then immediately running an incremental cook with no changes takes much longer than I would expect

In the logs:

Packages Cooked: 11911, Packages Incrementally Skipped: 45597, Packages Skipped by Platform: 442, Total Packages: 57950

So it is running the incremental.

I’m not sure why 20% of the project seems to require recooking from run-to-run (non-deterministic?)

and I’m not sure why doing the incremental cook is only 4 minutes faster than running it clean (28 minutes to 24 minutes)

Are these results expected? How should I go about debugging issues with this?

Matt.Peters · September 16, 2025, 11:47pm

Caveat: There may be a lot of work required to get incremental cook working well in 5.6. We have made a lot of improvements during the development of 5.7. If you don’t have the time to fix the problems with it, you may want to wait until 5.7 comes out, and in the mean time use MultiProcessCook as an accelerator of full recooks.

Cooking many packages is expected in 5.6 and 5.7, because we turn off by default the ability to incrementally cook packages that use types from your project rather than solely engine types. We turn them off by default because project types might contain hidden dependencies on files, managers containing raw pointers to packages, or other types of inputs that we don’t automatically detect. If any such undiscovered hidden dependencies exist, then updating those inputs would cause a false incremental skip and create an invalid build containing stale data. We are detecting and fixing these problems in engine types (we have fixed 95% of them in 5.6, and 99% of them in 5.7), but with project types that we don’t have the ability to test we have to turn them off to remain conservative. We don’t yet have a plan for how to improve this, but we know we need to develop one.

In the meantime, you can do the same as we do internally, and turn on allow incremental cook for classes in your project. An example of doing this is in the Lyra project:

Samples\Games\Lyra\Config\DefaultEditor.ini

[CookSettings]
+IncrementalClassScriptPackageAllowList=Allow,<ProjectRoot>

See also the notes in Engine\Config\BaseEditor.ini, for the keys IncrementalClassScriptPackageAllowList,IncrementalClassAllowList,IncrementalClassDenyList under CookSettings, for more detailed enable settings.

Part of our ability to allow Engine and internal project types to cook incrementally is to validate that there are no false incremental scripts, using a job that we run on the farm. This job is called incremental validate, and is implemented as a build graph task, executable by RunUAT.bat, in Engine/Build/Graph/Tests/IncrementalValidate.xml . When run, it does an incremental cook in the current workspace (you must preserve this workspace between runs for the test to be able to run), and uses the existing -diffonly feature of the cook (used to detect non-determinism) to cook anyway all of the skipped packages, and detect and report differences as errors if they have different results than the previous cook.

If you enable IncrementalClassScriptPackageAllowList for your types and are still seeing many packages cooked after no changes (we call these false recooks), then let me know and I can help you debug it. We have some diagnostic tools coming in 5.7 (preview release coming soon) which will help track them down. The reasons for false recooks in 5.6 are usually indeterminism in the calculations of dependencies.

I am surprised that the incremental cook skipped 45k packages out of 57k but still took 24 out of 28 minutes to run. Possibly the run time is dominated by some expensive packages you have that use your types, such as maps, and allowing your types to cook incrementally will fix that performance issue. Possibly you have expensive engine startup or expensive cookbythebookfinished operations which run every invocation of the cook and are not skipped. Some licensees report long times in calculating chunks at the end of the cook for example. And possibly the overhead of calculating dependencies for whether packages can be incrementally skipped is so expensive that it eats into the runtime benefits of the cook. To diagnose which of those issues it is, try capturing a trace, and possibly an off-the-shelf profile, of the cook. To capture a trace, run the CookCommandlet with -trace=default -tracefile=<PathToOutputFile>, and then run UnrealInsights to visualize the trace. https://dev.epicgames.com/documentation/en\-us/unreal\-engine/unreal\-insights\-in\-unreal\-engine. For an off-the-shelf profile, we recommend Event Tracing for Windows, visualized using Superluminal.

Let me know what you find; I’m interested in debugging issues with incremental cook for licensees.

Matt.Peters · September 19, 2025, 7:57pm

The performance cost of those unsolicited package warnings is nearly insignificant; I have changed the message in 5.7 to describe the actual issue they cause:

	TEXT("UnexpectedLoad: Package %s was loaded outside of any cook operation on a referencer package; we don't know why it was loaded. ")
	TEXT("Adding it to the current cook, but it will possibly not be found in future incremental cooks."),
 
	TEXT("UnexpectedLoad: TargetPackage %s was unexpectededly loaded during the cook of SourcePackage %s; it was not declared as a hard dependency (aka Import) of the SourcePackage. ")
	TEXT("We have to conservatively add this package to the cook, but it might not be needed at runtime. ")
	TEXT("Declare the TargetPackage as an import during editor save of the SourcePackage, or mark up its load with FCookLoadScope to specify whether it is runtime or editoronly."),

I’m surprised taking the fix from the hidden dependencies thread removed the false incremental recooks; I thought the impact of the bug would be to add unnecessary files to the cook and I didn’t think it would affect whether we could recook those files.

We should still fix those problems, though, especially if they are caused by a bug in the handling of instances rather than by an manual load from c++.

Can you add instrumentation to hit a conditional breakpoint in FPackageTracker::OnCreatePackage in Engine\Source\Editor\UnrealEd\Private\Cooker\PackageTracker.cpp when PackageName is the targetpackage (/Game/Env/Arch/Wood_Sm_50CM_01A_SM in the second message you posted), and report the callstack?

If the callstack for the load of Wood_Sm_50CM_01A_SM does not show any manual load of that package, then it must be another bug in the instance handling and I will need to work with you to gather some more data to diagnose it.

Those problems aside, is the performance of the incremental cook better now that it is has only 800 recooks?

Matt.Peters · September 23, 2025, 6:41pm

I have reproduced the UnexpectedLoad messages coming from FWorldPartitionLevelHelper::LoadActorsInternal. It occurs when WorldPartition loads a level instance it will embed into main level; the level instance is a SoftObjectPath and the cooker doesn’t know that WorldPartition needs to load it. The unexpectedLoad is having the predicted negative consequence in this case - the embedded package is added to the staged build sent to endusers even though it doesn’t need to be - and is therefore a waste of disk space for end users. Not a huge problem, but more significant than just a spurious warning. Fixing it requires tracking a new state in the cooker - packages that are embedded and need to have their dependencies added to the cook but don’t need to be added to the cook themselves. I will try to fix it for 5.7, but 5.7 deadline is soon and it might slip into 5.8.

I think that’s the last remaining issue you’ve mentioned; let me know if there’s still one that I’m overlooking, and let me know if you have any further problems with incremental cook. I’ll update this ticket when I submit the fix for the tracking of embedded packages.

anonymous-edc · September 17, 2025, 10:23pm

Hey Matt,

Thanks for the info.

So I grabbed the changelists from [Content removed]

as well as the DefaultEditor changes here. The results were noticeable.

In a cook immediately following another cook:

Packages Cooked: 800, Packages Incrementally Skipped: 58933, Packages Skipped by Platform: 493, Total Packages: 60226
 
Execution of commandlet took: 5m 31s (331.32 seconds)

However, I’m still seeing a bunch of

Unsolicited package %s was loaded outside of any cook operation on a referencer package; we don't know why it was loaded. Adding it to the current cook, but it will possibly not be found in future incremental cooks.Which seem to be assets we have as PrimaryAssetTypes in DefaultGame

and

Unsolicited package /Game/Env/Arch/Wood_Sm_50CM_01A_SM was loaded by package Test_MAP. This is a hidden dependency, causes poor cook performance, and it might be a bug to add it to the cook. Declare this package in the AssetRegistry dependencies of the loading package, or mark up its load with FCookLoadScope to specify whether it is runtime or editoronly.Which seems to be a static mesh actor in a LevelInstance placed on a WorldPartition MAP

the LevelInstance being Loot_Farm_Wheelbarrow_01_LI, which contains the Wood_Sm_50CM_01A_SM static mesh

[Image Removed]

anonymous-edc · September 22, 2025, 5:48pm

 	UnrealEditor-UnrealEd-Win64-Debug.dll!UE::Cook::FPackageTracker::OnCreatePackage(FName PackageName) Line 204	C++
 	UnrealEditor-UnrealEd-Win64-Debug.dll!UE::Cook::FPackageTracker::NotifyUObjectCreated(const UObjectBase * Object, int Index) Line 166	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FUObjectArray::AllocateUObjectIndex(UObjectBase * Object, EInternalObjectFlags InitialFlags, int AlreadyAllocatedIndex, int SerialNumber, FRemoteObjectId RemoteId) Line 321	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!UObjectBase::AddObject(FName InName, EInternalObjectFlags InSetInternalFlags, int InInternalIndex, int InSerialNumber, FRemoteObjectId InRemoteId) Line 259	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!UObjectBase::UObjectBase(UClass * InClass, EObjectFlags InFlags, EInternalObjectFlags InInternalFlags, UObject * InOuter, FName InName, int InInternalIndex, int InSerialNumber, FRemoteObjectId InRemoteId) Line 164	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!StaticAllocateObject(const UClass * InClass, UObject * InOuter, FName InName, EObjectFlags InFlags, EInternalObjectFlags InternalSetFlags, bool bCanRecycleSubobjects, bool * bOutRecycledSubobject, UPackage * ExternalPackage, int SerialNumber, FRemoteObjectId RemoteId, FGCReconstructionGuard * GCGuard) Line 3881	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!StaticConstructObject_Internal(const FStaticConstructObjectParameters & Params) Line 4981	C++
>	UnrealEditor-CoreUObject-Win64-Debug.dll!NewObject<UPackage>(UObject * Outer, FName Name, EObjectFlags Flags, UObject * Template, bool bCopyTransientsFromClassDefaults, FObjectInstancingGraph * InInstanceGraph, UPackage * InExternalPackage) Line 1921	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FAsyncPackage2::CreateUPackage() Line 10727	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FAsyncPackage2::InitializeLinkerLoadState(const FLinkerInstancingContext * InstancingContext) Line 6483	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FAsyncPackage2::ImportPackagesRecursiveInner(FAsyncLoadingThreadState2 & ThreadState, FIoBatch & IoBatch, FPackageStore & PackageStore, FAsyncPackageHeaderData & Header) Line 6423	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FAsyncPackage2::ImportPackagesRecursive(FAsyncLoadingThreadState2 & ThreadState, FIoBatch & IoBatch, FPackageStore & PackageStore) Line 6457	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FAsyncLoadingThread2::FinishInitializeAsyncPackage(FAsyncLoadingThreadState2 & ThreadState, FAsyncPackage2 * AsyncPackage) Line 4872	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FAsyncPackage2::ProcessLinkerLoadPackageSummary(FAsyncLoadingThreadState2 & ThreadState) Line 6832	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FAsyncPackage2::Event_ProcessPackageSummary(FAsyncLoadingThreadState2 & ThreadState, FAsyncPackage2 * Package, int __formal) Line 7340	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FEventLoadNode2::Execute(FAsyncLoadingThreadState2 & ThreadState) Line 5725	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FAsyncLoadEventQueue2::ExecuteSyncLoadEvents(FAsyncLoadingThreadState2 & ThreadState) Line 5931	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FAsyncLoadingThread2::ProcessAsyncLoadingFromGameThread(FAsyncLoadingThreadState2 & ThreadState, bool & bDidSomething) Line 9249	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FAsyncLoadingThread2::TickAsyncThreadFromGameThread(FAsyncLoadingThreadState2 & ThreadState, bool & bDidSomething) Line 9986	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FAsyncLoadingThread2::TickAsyncLoadingFromGameThread(FAsyncLoadingThreadState2 & ThreadState, bool bUseTimeLimit, bool bUseFullTimeLimit, double TimeLimit, TArrayView<int const ,int> FlushRequestIDs, bool & bDidSomething) Line 9605	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FAsyncLoadingThread2::FlushLoading(TArrayView<int const ,int> RequestIDs) Line 11281	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FlushAsyncLoading(TArrayView<int const ,int> RequestIds) Line 362	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!FlushAsyncLoading(int RequestId) Line 331	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!LoadPackageInternal(UPackage * InOuter, const FPackagePath & PackagePath, unsigned int LoadFlags, FLinkerLoad * ImportLinker, FArchive * InReaderOverride, const FLinkerInstancingContext * InstancingContext, const FPackagePath * DiffPackagePath) Line 1794	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!LoadPackage(UPackage * InOuter, const FPackagePath & PackagePath, unsigned int LoadFlags, FArchive * InReaderOverride, const FLinkerInstancingContext * InstancingContext, const FPackagePath * DiffPackagePath) Line 2158	C++
 	UnrealEditor-CoreUObject-Win64-Debug.dll!LoadPackage(UPackage * InOuter, const wchar_t * InLongPackageNameOrFilename, unsigned int LoadFlags, FArchive * InReaderOverride, const FLinkerInstancingContext * InstancingContext) Line 2134	C++
 	UnrealEditor-Engine-Win64-Debug.dll!FWorldPartitionLevelHelper::LoadActorsInternal(FWorldPartitionLevelHelper::FLoadActorsParams && InParams, FWorldPartitionLevelHelper::FLoadedPropertyOverrides && InLoadedPropertyOverrides) Line 933	C++
 	UnrealEditor-Engine-Win64-Debug.dll!FWorldPartitionLevelHelper::LoadActorsWithPropertyOverridesInternal(FWorldPartitionLevelHelper::FLoadActorsParams && InParams) Line 515	C++
 	UnrealEditor-Engine-Win64-Debug.dll!FWorldPartitionLevelHelper::LoadActors(FWorldPartitionLevelHelper::FLoadActorsParams && InParams) Line 623	C++
 	UnrealEditor-Engine-Win64-Debug.dll!UWorldPartitionRuntimeLevelStreamingCell::OnPrepareGeneratorPackageForCook(TArray<UPackage *,TSizedDefaultAllocator<32>> & OutModifiedPackages) Line 437	C++
 	UnrealEditor-Engine-Win64-Debug.dll!UWorldPartition::PrepareGeneratorPackageForCook(IWorldPartitionCookPackageContext & CookContext, TArray<UPackage *,TSizedDefaultAllocator<32>> & OutModifiedPackages) Line 94	C++
 	UnrealEditor-Engine-Win64-Debug.dll!FWorldPartitionCookPackageSplitter::PopulateGeneratorPackage(ICookPackageSplitter::FPopulateContext & PopulateContext) Line 157	C++
 	UnrealEditor-UnrealEd-Win64-Debug.dll!UE::Cook::FGenerationHelper::TryCallPopulateGeneratorPackage(TArray<ICookPackageSplitter::FGeneratedPackageForPopulate,TSizedDefaultAllocator<32>> & InOutGeneratedPackagesForPopulate) Line 1189	C++
 	UnrealEditor-UnrealEd-Win64-Debug.dll!UCookOnTheFlyServer::BeginCacheObjectsToMove(UE::Cook::FGenerationHelper & GenerationHelper, UE::Cook::FCookGenerationInfo & Info, UE::Cook::FCookerTimer & Timer, TArray<ICookPackageSplitter::FGeneratedPackageForPopulate,TSizedDefaultAllocator<32>> & GeneratedPackagesForPopulate) Line 3775	C++
 	UnrealEditor-UnrealEd-Win64-Debug.dll!UCookOnTheFlyServer::PrepareSaveGenerationPackage(UE::Cook::FGenerationHelper & GenerationHelper, UE::Cook::FPackageData & PackageData, UE::Cook::FCookerTimer & Timer, bool bPrecaching) Line 3662	C++
 	UnrealEditor-UnrealEd-Win64-Debug.dll!UCookOnTheFlyServer::PrepareSaveInternal(UE::Cook::FPackageData & PackageData, UE::Cook::FCookerTimer & Timer, bool bPrecaching, UE::Cook::ESuppressCookReason & OutDemotionRequestedReason) Line 4264	C++
 	UnrealEditor-UnrealEd-Win64-Debug.dll!UCookOnTheFlyServer::PrepareSave(UE::Cook::FPackageData & PackageData, UE::Cook::FCookerTimer & Timer, bool bPrecaching, UE::Cook::ESuppressCookReason & OutDemotionRequestedReason) Line 4064	C++
 	UnrealEditor-UnrealEd-Win64-Debug.dll!UCookOnTheFlyServer::PumpRuntimeSaves(UE::Cook::FTickStackData & StackData, unsigned int DesiredQueueLength, int & OutNumPushed, bool & bOutBusy) Line 4976	C++
 	UnrealEditor-UnrealEd-Win64-Debug.dll!UCookOnTheFlyServer::PumpSaves(UE::Cook::FTickStackData & StackData, unsigned int DesiredQueueLength, int & OutNumPushed, bool & bOutBusy) Line 4864	C++
 	UnrealEditor-UnrealEd-Win64-Debug.dll!UCookOnTheFlyServer::TickMainCookLoop(UE::Cook::FTickStackData & StackData) Line 1574	C++
 	UnrealEditor-UnrealEd-Win64-Debug.dll!UCookOnTheFlyServer::TickCookByTheBook(const float TimeSlice, ECookTickFlags TickFlags) Line 1420	C++
 	UnrealEditor-UnrealEd-Win64-Debug.dll!UCookCommandlet::RunCookByTheBookCook(UCookOnTheFlyServer * CookOnTheFlyServer, void * StartupOptionsAsVoid, ECookByTheBookOptions CookOptions) Line 865	C++
 	UnrealEditor-UnrealEd-Win64-Debug.dll!UCookCommandlet::CookByTheBook(const TArray<ITargetPlatform *,TSizedDefaultAllocator<32>> & Platforms) Line 626	C++
 	UnrealEditor-UnrealEd-Win64-Debug.dll!UCookCommandlet::Main(const FString & CmdLineParams) Line 270	C++

Hey Matt,

Here’s the callstack. Looks like we’re in LoadPackage.

The performance is better with the project classes now also getting incrementally cooked.

I'm surprised taking the fix from the hidden dependencies thread removed the false incremental recooks; I thought the impact of the bug would be to add unnecessary files to the cook and I didn't think it would affect whether we could recook those files.I don’t think this had a noticeable impact. I just put the information here for completeness of changed variables.