Cook warning messages "Package was added to the runtime discovery queue after starting BuildDependencies phase." in an up-to-date cook

Development Pipeline & Plugins
, ,
1

In our project we heavily use multiprocess cooking together with iterative cooking (legacyiterative).

After upgrading to UE 5.6 we now noticed following warning messages being printed in our cooker “Package was added to the runtime discovery queue after starting BuildDependencies phase”. They occur when doing an iterative + multiprocess cook right after having done a full cook, so essentially all assets are up-to-date and there is no work to be done by the child cookers. All the packages for which this warning is being printed appear to be from the EInstigator::StartupPackage set.

I found that in CL 38852986 Epic attempted to address this issue by adding a PhaseTransitionFence which ensures that parent cooker waits for child cookers to send any and all their messages before it proceeds to the ECookPhase::BuildDependencies phase. I noticed that this fence wasn’t being waited on properly because the very first fence we send, which is sent using the heartbeat system, has a value of 0 and the FCookWorkerServer::LastReceivedHeartbeatNumber being also initialized to 0 means that we never wait on this very first fence. Initializing the FCookWorkerServer::LastReceivedHeartbeatNumber to -1 addresses this fence issue, and fixes the warning messages.

One side effect of this is that a fully up-to-date cook now has to wait for the child cookers to initialize and connect to the parent before shutting everything down, which seems add about 20sec of additional overhead for us in a fully up-to-date cook.

Can you let us know if this is an acceptable solution?

2

Steps to Reproduce

  1. Run a full cook for any platform.
  2. Run a second cook with multiprocess cook and iterative cook (legacyiterative) enabled, which should result in assets being up-to-date.
3

Yes, that’s the way the LastReceivedHeartbeatNumber should have been written, thanks for the fix.

For the 20 second performance degredation, it is an edge case that we didn’t consider when adding these phases; we could add a special case to skip it in the case that no packages have been assigned to any worker.

4

Thank you for the quick reply. I’ll submit the fix on our end. I do think it would be worthwhile to speed up the use case where there is no work assigned to the child cookers, 20 seconds is a long chunk of time.

5

I’ve put that on my backlog for a quick fix in the next few weeks, but I might not be able to get to it, in which case I will add it to the backlog, no guarantee of ETA.

6

I added the optimization, along with another bugfix for the phase transition, in CL 45117531, aka github commit a622953f734857b9f9e7229d36542facc8f954fe. It looks like the optimization is minor, though. After skipping the wait for the CookWorkers and passing through the phase transition, the Cooker still waits for the CookWorkers to finish loading and shutdown, in FCookDirector::PumpCookComplete. Skipping that shutdown was more complicated or riskier, so I didn’t implement it. Maybe you’re not seeing that because the time it takes for wait for the CookWorkers to startup at that point is hidden by other work that has to be done on the director