Incremental Cook of World Partition World Misses OFPA File Dependencies

Development Pipeline & Plugins
, ,
1

Hello!

Recently, we’ve started using Incremental Cook to improve our atrocious cook times. We’re on 5.6.1 but have taken many changesets from 5.7 and 5.8 around cook indeterminism, so there’s lots of “future code” here. However, we’ve noticed one case which completely breaks our Incremental Cooks. The specifics are hard to pin down, but here’s what we’ve observed and understand about the problem.

The core symptom is that, occasionally, cooks will begin failing randomly after Incrementally Cooking multiple times in a row successfully. They will fail with errors of this structure:

LogCook: Error: Content is missing from cook. Source package referenced an object in target package but the target package was marked NeverCook or is not cookable for the target platform.
  Source package: <Some World Partition Generated Cell>
  Target package: <Some On-Disk Asset, Like a Mesh or Texture>
  Referenced object: <Some On-Disk Asset, Like a Mesh or Texture>

Notably, it is often large numbers of these appearing suddenly for the same World Partition Generated Cell. For example, we went from a valid cook to an incremental cook which was missing almost 30 items from the cell “MainGrid_L0_X0_Y0_DL69955158” in one map. It’s not always data layer cells or any other detectable pattern in terms of which kind of cell is affected. These worlds are using Runtime Hash Set partitioning, multiple streaming grids, and the error does not seem to be related to embedded level instances or subworlds.

Looking in the map and adding additional logging confirms the missing content is seemingly always a reference of some OFPA actor in the world. For example, we have seen this with a skeletal mesh “SK_Human_Tops_HikingBackpack_Straps_01” which was only included in a map due to a SkeletalMeshActor placed in the world.

Moreover, this only seems to occur after the asset dependency (but NOT the OFPA actor) are modified. So in the previous example, we found this started occurring after one of our artists had updated the SK_Human_Tops_HikingBackpack_Straps_01 asset. The OFPA actor in the map wasn’t touched -- it didn’t need to be, after all -- but our next incremental cook failed in this way.

Repeating the Incremental Cook always fails in the same way because it’s just replaying from the Zen Oplog for the affected assets. We performed a full cook after, which resolved the issue.

So, my best guess that OFPA actor downstream dependencies are not correctly accounted for in the artifact key determination to see if a World Partition world / cell needs recooked? But I’m guessing here.

Can you give us any guidance for how to further debug this? I can somewhat reliably reproduce this and get additional information if needed.

Thanks!

[Attachment Removed]

2

Steps to Reproduce

  1. Create a World Partition map
  2. Add some OFPA actors to it
  3. Incrementally cook to establish a baseline
  4. Modify the OFPA actor dependencies (i.e., update the Static Mesh file on disk referenced by a Static Mesh Actor without modifying the Static Mesh Actor itself)
  5. Incrementally cook again
  6. Observe the error “LogCook: Error: Content is missing from cook. Source package referenced an object in target package but the target package was marked NeverCook or is not cookable for the target platform.” for the specified OFPA actor
    [Attachment Removed]
3

This sounds like the problem which was recently reported and we fixed in 6.0 code:

* WorldPackage has a package builddependency on each of its actors.

* The actors have builddependencies on the data they rely on, but these dependencies are not included in the WorldPackage’s dependencies because the WorldPackage did not specify transitive build dependency on each of its actors, it only specified a non-transitive package dependency.

* The generated package picks up those dependencies and would rebuild if asked, but we never check whether it needs to rebuild because we didn’t rebuild the WorldPackage and so we skip testing whether we need to rebuild the generated packages.

This was fixed in github commit 0a2d472ebae920dfacef1f4fb24527b639fe9221

> IncrementalCook: Add transitive dependencies from a Generator to its GeneratedPackages, so that when a GeneratedPackage needs to recook, the Generator sees that and recooks itself so it can process it. This changes the previous design plan for …

That fix integrates fairly easily into 5.7, but I’m not sure how well it will work in 5.6. I believe there were several fixes for transitive build dependencies in 5.7.

[Attachment Removed]

4

Thanks, this seems promising. I’ll try to integrate and see if it changes anything for us. This will probably takes several days to meaningfully test, but I’ll follow up once I feel confident either way.

[Attachment Removed]

5

This is looking good for us. After a week of regular incremental cooking, we have seen no drift from the OFPA files.

The merge was actually pretty simple. Other than some logging macro changes to a format we don’t have yet, it merged cleanly. I will review the other transitive build fixes you mention to see if we’re missing any functionally-critical changes, but otherwise this seems good! I’ll roll it out to the rest of my team and -- if we’re seeing consistently stable cooks -- mark this answered. Thank you!

[Attachment Removed]