In researching incremental cook with 5.7 and Cooked Data Snapshots, we’re wondering if it’s a valid use case currently to use cooked data snapshots as a base for incremental cooking. In other words, if our CI system cooks on CL 1000 and publishes a snapshot, can I then pull down that snapshot, sync to CL 1001, and use -cookincremental locally to only cook incrementally the 1 CL difference in files (assuming other caveats of incremental cooking apply and that the files in CL 1001 are all valid for incremental cook)?
In digging through the snapshot code and the incremental cook code, I have a few questions (assuming this is valid at all):
- Does the snapshot need to be produced with an incremental cook? Since the first incremental cook builds oplog data needed for future incremental cooks, I’m assuming this is a requirement. If a normal full cook (non-incremental) was used to produce the snapshot, am I able to download the snapshot and cook incrementally off of that? Or will my cook be a full cook since the snapshot didn’t contain the base data?
- If the snapshot does need to be incremental, is there a way to force a baseline incremental cook? The goal here is to avoid incremental drift on the CI machine producing the snapshot, so I want every snapshot it produces to be effectively a full cook, but with the extra data needed for an incremental cook on top of it. I might be misunderstanding it, but I don’t see a way to essentially do a “clean incremental cook” (i.e. specify both -cookincremental and -forcerecook together). It seems like these two params would interfere with each other since cookincremental directly sets bForceRecook = false.
If this is all possible to do, are there any other gotchas or caveats I should be aware of? I know that incremental cooking is still only Beta in 5.7, but it seems stable enough to use for us in some limited cases. We were hoping to be able to provide regular snapshots that people could use as a base.
[Attachment Removed]
Yes, using snapshots for incremental cooking is an intended use and we are testing it on projects within Epic in 5.8. We have not yet hardened it however, especially not in 5.7, so you may encounter errors with it.
The baseline cook done by the CI system does not need to be an incremental cook; it can be a full recook. It does need the config variable that activates storage of cook dependencies to be set, but this variable is set by default in 5.7. (This required config value is Editor.ini:[CookSettings]:bLegacyIterative=false; you can find its default value set to false in Engine\Config\BaseEditor.ini). The baseline cook must also be cooked with bUseIoStore=true and bUseZenStore=true in Game.ini:[/Script/UnrealEd.ProjectPackagingSettings]. If the cooker is correctly configured to activate the storage of cook dependencies, you will see this log message in the cookcommandlet’s log:
LogCook: Display: INCREMENTAL COOK DEPENDENCIES: Enabled. Incremental Cook Dependencies are enabled in Editor.ini:[CookSettings]:LegacyIterative=false.Otherwise you will see INCREMENTAL COOK DEPENDENCIES: Disabled… and there will be more information about why it was disabled.
Note that the sequence of commands requires oplog-snapshot in addition to oplog-export and oplog-import:
- (Build machine) UnrealEditor <Project> -run=cook -targetplatform=<Platform>
- -forcerecook=true to force a recook is optional, depending on whether you want a full recook
- (BuildMachine) Engine\Binaries\Win64 zen.exe oplog-snapshot --project <ProjectId> --oplog <PlatformName>
- This copies the non-uasset local files referenced by the oplog to be copied into zenserver so they are available in the export
- (BuildMachine) Engine\Binaries\Win64 zen.exe oplog-export --project <ProjectId> --oplog <PlatformName> …
- You will also need to specify -zen or -file for where to export
- (UserMachine) Engine\Binaries\Win64 zen.exe oplog-import --project <ProjectId> --oplog <PlatformName> …
- You will also need to specify -zen or -file that matches where the export was stored
Some of the issues we have fixed with this workflow in 5.8 include indeterminism in the config settings that always trigger a full recook. e.g. spurious differences in CoreRedirects between the build machine and the user machine, due to different sets of plugins enabled. I don’t think any of those will be relevant to the average project, but it is possible you will hit them. In that case you will get spurious recooks the first time you cook on top of a snapshot created by the build machine and you will need to use our incremental cooking false recook tools to diagnose them.
Other than false recooks, I don’t know of any caveats with using this workflow in 5.8; there may still be some in 5.7 that I am forgetting about. Let me know if you run into any issues.
[Attachment Removed]
Thanks for the reply. It looks like we had made some incorrect assumptions about this. What you mentioned appears to be correct, that we don’t need to use an incremental cook as the base. I actually have some more questions about the process of exporting and importing the oplog, but I’ll start a new question on it since they’re unrelated to the original topic here.
[Attachment Removed]