Sorry for the delay in this response. We are publishing a user guide for 5.7 that I think will answer all of these question; I’ll quote some excerpts from its rough draft. Note that we have made many robustness improvements to Incremental Cook between 5.6 and 5.7, so you should try to integrate to 5.7 as soon as possible to get those improvements. Even without those improvements however, Incremental Cook in 5.6 is much more robust thatn LegacyIterative, so you should use Incremental Cook if you can get it working performantly.
> To be fully activated on a project, IncrementalCook must be configured to allow assets of native C++ classes defined by the project (or the project’s plugins) to be Incrementally Skipped. We turn this off by default because project types might contain Hidden Dependencies on other packages, files, managers containing raw pointers to packages, or other types of inputs that we don’t automatically detect. If any such Hidden Dependencies exist, then updating those inputs would cause a False Incremental Skip and create an invalid build containing stale data.
How many packages do you have that are incrementally skipped when you cook back to back? Is it possible that the 2500 packages recooked every time are caused by this by-default disallowal of your native c++ types?
I don’t think the use of __Generated__ content folders is likely to be the cause of the packages not being skippable, it’s more likely to be caused by the missing config setting, which disproportionately affects map packages because map packages almost always include project-specific types.
I have copied the how-to section of the user guide for enabling your native types in the list below. Before that, some text explaining Hidden Package Dependencies. The how-to section has instructions for setting up TObjectPtr and analyzing your c++ types that are somewhat time-consuming, and the actual enablement just requires adding a single line to config. This text gives the explanation for why the TObjectPtr setup and validation is important. But depending on how complex your types are and what your tolerance is for failure, you can chose to skip these steps to start with and allow your types to be incrementally skipped without them, at the cost of possible stale data when hidden dependencies change.
> Correctness of Incremental Cook is a primary concern. Correctness for Incremental Cook is defined as having the same results as a cook would have if there was no previous cook and every package was an Incremental Newcook. Some Incremental Cook errors are unrelated to the data stored for each package, such as the incorrect compilation of non-package cook artifacts (AssetRegistry, ShaderLibraries, Metadata, licensee-specific artifacts). But the commonly occurring errors are a difference in the TargetDomain bytes or metadata of a previously cooked package from the bytes or metadata that would be created if it were Incrementally NewCooked. If a package is found to be Incrementally Stable and is therefore Incrementally Skipped, but if recooked its TargetDomain bytes or metadata would necessarily be different, the Incremental Skip is an error called a False Incremental Skip. False Incremental Skips are caused by an incomplete set of dependencies for the package.
> There are many kinds of dependencies a package can have that can affect the cooked bytes or the metadata stored for the package; a partial list of these:
- PackageDependency: Values stored in a Target package on disk that was saved by the editor are read during load/save of the cooked version of a Source package and affect the bytes that are written.
- Every TargetDomain package automatically has a PackageDependency on its WorkspaceDomain package.
- PackageDependencies for other packages are automatically collected, for types that support it, by tracking of TObjectPtr dereferences (see below).
- NativeClassDependency: A change to a native UClass causes a change in packages that have instances of (or otherwise depend on) that UClass.
- The list of classes used by objects in the package are automatically collected, other classes can be added by system-specific code.
- Changes to each native UClass that are exposed to Unreal’s reflection system (UPROPERTY, UCLASS, other fields managed by UnrealHeaderTool) are automatically collected and changes in them detected.
- Changes to C++ hooks used by a class (e.g. native UObject::Serialize overrides) are not automatically detected and the programmer making changes to them must bump a version in the class’s AppendToClassSchema function.
- Changes to non-native classes such as BlueprintGeneratedClasses are not detected by NativeClassDependency and instead are detected by a PackageDependency on the package containing the Blueprint.
- ConfigDependency and ConsoleVariableDependency: Values read from ini, or from overrides on the CookCommandlet command line.
- These are automatically collected by instrumentation in the config system.
- Some C++ code can bypass the automatic collection and the authors of that code then need to declare them manually.
How-to Section from the user guide:
> Setting Up Incremental Cook
- Allow your Project’s native C++ types to be incrementally skipped.
- This may take some time to implement; you can skip this step and still gain IncrementalCook performance benefits from assets in your projects that only use already-validated Engine types, such as UTexture2D, UMaterial, and UStaticMesh.
- Modify all of your native classes to use TObjectPtr rather than raw UObject* properties and (when feasible) function parameters.
- TObjectPtr was written to function as a drop-in replacement for UObject* and does not have significant performance costs (with a small but noticeable exception when we subscribe to its updates during the cook for dependency tracking).
- This change is mostly mechanical and we have a script to procedurally change your c++ code to make it. Running that script, compiling, and fixing a few (or none) compile errors for edge cases should be sufficient to make the change.
- See the section on TObjectPtr in the UE5 Migration Guide: https://dev.epicgames.com/documentation/en-us/unreal-engine/unreal-engine-5-migration-guide
- Statically analyze your classes that are likely to have hidden dependencies, and add code to declare them.
- Loading non-Unreal-package files.
- Reading UObject* gathered by a manager class not using TObjectPtr.
- Caching the value of ConfigVariables or ConsoleVariables in function static variables, so that our automatic detection does not pick them up.
- Classes with native Serialize/PostLoad/PreSave/OnCookEvent/BeginCacheForCookedPlatformData functions that are frequently changed. You will need to add an AppendToClassSchema function to your UObject subclass, and add a version number or guid to it, and instruct your C++ developers to bump that version when they make changes.
- For how to declare dependencies, see <TODO: LinkToUnrealEngineC++CookingAPI>
- Enable IncrementalCook of your types in config
- Editor.ini:[CookSettings]:IncrementalClassScriptPackageAllowList
- Add a line +IncrementalClassScriptPackageAllowList=Allow,<ProjectRoot>
- Add the text “<ProjectRoot>” as a literal string, do not replace it with your project root. This string is interpreted by the cooker and replaced with all script packages in your Project or project plugins.
- See Samples\Games\Lyra\Config\DefaultEditor.ini for an example
- See the notes in Engine\Config\BaseEditor.ini, for the keys IncrementalClassScriptPackageAllowList,IncrementalClassAllowList,IncrementalClassDenyList under [CookSettings], for more detailed enable settings.
- You may want to enable most of your types and disable specific problematic ones.
For your question of the problems with legacy “Modified Only”:
> In previous versions of Unreal, the mode of cooking that attempted to skip work of previously cooked packages had another name: Iterative Cook. We changed the name from Iterative Cook to Incremental Cook to indicate the improvement in functionality, and we now refer to that Iterative cook system as LegacyIterative. LegacyIterative is still available when selected by config, or when requirements for IncrementalCook (such as the use of ZenServer for storage of cook results) are not met. The primary failing of LegacyIterative cook was that it did not include NativeClassDependencies, so any change to any class would cause False Incremental Skips and the stale values would frequently cause the runtime to crash. LegacyIterative also had less robust detection of PackageDependencies, causing False Incremental Skips due to PackageDependencies that were Hidden Dependencies. LegacyIterative also had a coarser detection of ConfigDependencies, causing a performance problem - False Incremental Recooks.
[Attachment Removed]
