Deterministic Builds and Cooking

Sep 16, 2020.Knowledge

Deterministic Cooking

When making a build of an updated version of a released project, it’s important that a cooked asset doesn’t change if the source asset doesn’t. The ability to cook the same source asset twice and get the same results means the cooking is deterministic. You would evaluate the determinism of your project by re-cooking and rebuilding the project twice without any changes to the source assets, and ideally the resulting package content would remain the same (or binary equivalent) to the previous build. Once you have a build with your patch (or DLC), the Switch Authoring Tool can be used to create a patch by comparing an updated build of your title against the originally submitted build.

Confirming your project cooks in a deterministic way:

To evaluate the determinism of your project, do the following:

  1. Start by doing a complete cook of your assets for your target platform.
  2. In your project folder, navigate to Saved > Cooked and make a copy of the target platform folder by renaming it or copying it to another directory.
  3. Start another complete cook of your assets.
  4. You can now do a binary file-by-file comparison of the first Saved > Cooked > target platform folder to the copy from the previous build.

What to do if you find an asset is not cooking in a deterministic way:

The first thing you should do if an asset is different in successive cooks is to re-save the asset. We have versions of asset formats that do change over time, necessitating code to upgrade older versions to new versions. This upgrade code is generally run on load, but if an asset is not saved, it will continue to run this upgrade code on load, potentially increasing loading times, as well as potentially producing different results on subsequent loads. There is a commandlet you can run to re-save all assets as well, which is worth doing from time to time.

If the asset is a custom one created for the project, if it has custom loading and saving code, it’s possible there’s some non-deterministic behavior in that code that needs to be changed.

If the asset is an engine-side built-in type, you may have to contact us on UDN for further support. In this case, it’s important to identify:

  1. If the issue happens with all assets of that type
  2. If the issue persists if you migrate the asset to a new project that can open with an unmodified version of the editor
  3. If the issue persists in the latest released version of Unreal Engine

Sources of non-deterministic cooking behavior:

There are several ways assets can start to cook in a non deterministic way.

  • Problems that Epic has control over:
    • Over time, changes to asset formats and serialization code on the engine side can inadvertently introduce these issues. While we actively test for and correct these issues when they come to our attention, our tests can’t cover all the content and use cases that developers that use Unreal Engine generate, so there may be issues we don’t catch before a given version of Unreal Engine is released. If you think you’ve found a deterministic cooking issue you think we have missed, please let us know.
  • Problems outside of Epic’s control:
    • One issue we have seen in the past is that some third party tools can cause deterministic cooking errors. One example was a code integration from a third party that added a globally unique identifier (GUID) to Primitive Components, but regenerated it each time. This caused level files to have deterministic cooking problems, and patch sizes to increase dramatically.
    • It’s possible for project side logic to cause deterministic cooking problems. This can be as simple as, for example, defining a blueprint node in C++ that has some non-deterministic behavior with a result that is stored in a serialized variable, and then calling that node from a blueprint construction script. Blueprint construction scripts get run during the cooking process (for instances of that blueprint placed in the editor as opposed to ones spawned at runtime), so if care is not taken, that behavior will differ with each cook, and the resulting assets will differ, so when making a patch, the patch size will be artificially inflated. The non-deterministic behavior may not be explicit in the project side code, but come from calling functions from an engine system that generate a GUID or have some other non-deterministic behavior under the hood. This can be avoided by marking any blueprint nodes defined in your project’s code that don’t need to be called in a construction script to be not place-able in construction scripts. If you do need to define construction script callable nodes in C++ in your project, do so with care. When creating them, test by creating an example blueprint that uses the node in a construction script and then cook that asset twice in a row with no changes to make sure it gives the same binary result. Writing custom serialization code in a project is another area where care has to be taken to maintain determinism across successive serializations.

Maintaining a deterministic build is ideal but does not always happen. Unreal Engine 4 attempts to detect and resolve these issues, which is an ongoing concern with deterministic builds, however, it is a good idea to check how deterministic your cooked asset build is and address any issues before submitting the initial release version of your title.