We are looking to move from iterative cooking over to incremental cooking. We are currently on UE 5.7.1. Our shared server where we export snapshots to is v5.7.6 and our local workstation zenservers are also 5.7.6.
The behavior I am seeing is that our game appears to be cooking incrementally pretty well. A full cook and then a follow up with -cookincremental generally recooks a small amount of package (144 in my current test) and takes about 3-4 minutes. The trouble comes in when we try to pull a cooked snapshot originating from our CI system for the same CL we are currently synced at, then run an incremental cook against it. In this case we are seeing a larger number of uassets needing cooking at the start of cook (5348) and is taking around 30 minutes. Both cooks are using multiprocess cooker.
This is the output of the locally incremental cook:
Packages Cooked: 144, Packages Incrementally Skipped: 0, Packages Skipped by Platform: 0, Total Packages: 144
This is the output of the incremntal cook on top of a imported snapshot (from same CL):
Packages Cooked: 5346, Packages Incrementally Skipped: 0, Packages Skipped by Platform: 2, Total Packages: 5348
There are some odd things about this in general. The first odd thing is those numbers don’t seem quite right to me, I see things in the log that make it seem that in both cases there are more packages being actually cooked than these numbers indicate but maybe the numbers don’t include the dependencies? The other odd thing is that a full cook of our project is over 130k packages total and takes around 30 minutes as well with the same multiprocess cook settings on the same hardware.
One thing I found along the way is that if you cookincremental unversioned on top of a versioned cook and vice versa it will always be a full cook which probably makes sense. Our CI agents always run versioned cooks so I swapped the client to also run a versioned cook to move around this so all these results are using versioned cooked output.
We are also using PSO Caching but as far as I can tell I don’t think that is coming into play here but I wanted to call it out just in case.
One final note is that I have tested this same process (using our same buildgraph calls) end to end between 2 different workstations where one workstation pushes the snapshot to the shared zenserver and the other pulls it down (synced to same CL) and runs a cookincremental and for some reason I still can’t sort out this seems to be working correctly (similar behavior to a local cookincremental) I’m not really far enough along to understand why this seems to work.