Cooking will cook maps not requested to be cooked

Development Pipeline & Plugins
, ,
1

We would like to reduce our cooking times. We use Project Launcher to build and cook our build. We don’t specify any MapsToCook in defaultgame.in. We set the maps inside the Project Launcher. When iterating on work, we usually would have this set to a single map. We also specify incremental cooking. What we end up seeing is basically all of our maps getting cooked. It appears that if some asset changed, let’s say a staticmesh, then any asset that referenced that staticmesh will also get cooked. This ends up doing an upside-down tree and reaches basically every map.

Ideally, we could set what map we would want to cook and only assets referenced by that map would be candidates for cooking. If the referenced asset was changed since the last cook, it would get cooked. If it was unchanged, it would be skipped. This would allow for the optimal iteration workflow.

In the past, we have tried using DirectoriesToNeverCook in the DefaultGame.ini and specified all the maps other than the one we want to cook. This can help, but it is not easy to maintain and it does create cook “errors”. We can work around the cook errors by cooking again, but still, not a great workflow to push to the team.

I’m looking for some advice on achieving my desired optimal iteration workflow.

2

What is your commandline for the cook? I think you are using -iterate, but want to confirm.

-iterate has been around for 10+ years. It has some flaws (e.g. it does not detect changes in the binary and does not recook packages based on the classes they are using changing in layout).

It also has the property that it cooks the previously cooked packages even if not requested in the currentcook:

Engine\Source\Editor\UnrealEd\Private\CookOnTheFlyServer.cpp

... void UCookOnTheFlyServer::PopulateCookedPackages(TArrayView<const ITargetPlatform* const> TargetPlatforms) ... // *) Cook all modified packages even if the requested cook packages don't reference them ... // cook on the fly will queue packages when it needs them, but for cook by the book we force cook the modified files // so that the output set of packages is up to date (even if the user is currently cooking only a subset) WorkerRequests->AddStartCookByTheBookRequest(FFilePlatformRequest(PackageData->GetFileName(), EInstigator::LegacyIterativeCook, TConstArrayView<const ITargetPlatform*>{ TargetPlatform }));

Possibly you could modify code locally to turn off that behavior.

In 5.6, we have published as an experimental feature the new version of iterative cooking that we have been working on, incremental cooking. It is at least as robust in 5.6 as it iterative cooking is in 5.6, but it requires enabling ZenServer as well. Incremental cook does not have that property of forcing the cook of previously cooked packages. It might be an option for you if turning off the behavior in -iterate is not. ZenServer and IncrementalCook are enabled via

`// ZenStore enabled in Game.ini:
[/Script/UnrealEd.ProjectPackagingSettings]
bUseZenStore=True

// CookIncremental enabled in commandline argument:
-cookincremental
-cookincremental -forcerecook // Forces a recook despite cookincremental being on

// The configuration to enable CookIncremental will change in 5.7`

I don’t know whether this property of -iterate is the cause of your problem; it would only manifest if you do a cook with all of your packages followed by a narrower -iterate cook of only your requested packages. If the problem occurs even without that earlier, wider cook, then you might instead have a case where even in non-iterative cooks the packages are getting pulled in. If that is the case, then you can find out what is pulling them into the cook by running the cookcommandlet with showinstigators on (via commandline argument -dpcvars=cook.displaymode=4):

UnrealEditor.exe <ProjectName> -run=cook -targetplatform=<PlatformName> -dpcvars=cook.displaymode=4When that display mode is on, the cooker will log an instigator message for each cooked package:

LogCook: Display: Cooking /Game/PackageName, Instigator: { AssetManagerModifyCook }

3

No, removing the AddStartCookByTheBookRequest from PopulateCookedPackages will not undercook; the packages that are used by your requested primaryassets will still be requested elsewhere, in UAssetManager::ModifyCook. The call in PopulateCookedPackages is only used to force the cook of packages that are not requested in the current cook but that were requested in a previous cook.

If you do a full recook by removing -iterativecooking (and -iterativedeploy), and in that full recook cook without the maps you do not want requested, do those maps still get cooked in the full recook?

4

For slow full cooks, you should try multiprocesscook: AdditionalCookArgs=“-cookprocesscount=N”, N > 1.

If your data is not too entagled, it will speed the cook up alot (not as much as xN, but maybe close). It uses a lot of memory though. You should experiment with different values for N.

I don’t think of anything in 5.5 that would have impacted how much gets deleted after a cook error, but maybe there’s some detail I’m not thinking of that would do it. There might be information in the log about it.

5

here are my relevant commandline args:

-build -cook -iterativecooking -iterativedeploy -stage -deploy

If -iterativecooking will cook force cook modified, that seems like the right thing. The file was modified, so it’s a candidate for cooking. But what you’re saying is it will also cook it even if it’s not referenced by a primary asset type, like a specified map. So if I removed that AddStartCookByTheBookRequest line wouldn’t that mean necessary assets wouldn’t get cooked?

And if I didn’t use -iterativecooking, wouldn’t that cook everything, changed or not. So if I’m basically seeing everything get cooked either way, and the only way to skip cooking some is to remove that AddStartCookByTheBookRequest call, which could then undercook, then I’m not sure what my options are.

I guess my options are to manually control DirectoriesToNeverCook and point it at as much as I can?

6

“force the cook of packages that are not requested in the current cook but that were requested in a previous cook.”

I don’t think I understand the value of cooking assets that were requested in a previous cook. Is that just in case that cook failed and left a bunch of uncooked assets?

My full cooks are huge. I had to delete my zen storage and cooked folder, etc basically start from scratch. This took 12 hours on my machine. So it’s difficult to experiment with the full cooks. I even added a Dialog to FLooseCookedPackageWriter::DeleteSandboxDirectory which is the thing that deletes the cooked folder just in case. The dialog asks if I want to allow it to delete. And it seems to want to do delete a lot. One tiny cook error, and it will nuke your cooked folder. We just moved to 5.5, and that feels new.

I will try AddStartCookByTheBookRequest on my next cook after getting latest and see what I get.