Hello !
As we are attempting to move our build farm to use multi-process cooks to reduce iteration between generated versions, we believe we are nearly there.
I have launched 12 single-process cooks of our game, where everything is stable in the World Partition logs under <ProjectName>\Saved\Logs\WorldPartition.
Then, I attempted to launch 12 multi-process cooks, but some World Partition Streaming Generation logs have some small variance where HLODRelevant can be falsely set to true, and its RuntimeHLODLayer can be assigned to something that should have been none. It always happens on instances of blueprints deriving from a PackedLevelActor.
Variance is very small, but seems to happen much more easily on very small World Partition worlds. Also, the variance always seems to happen when it is written not long after an initial dump of the World Partition log, so if I have three parallel cooks, I could have three logs of the same world.
I have also noticed that in the cook logs, there are many cook warnings, especially about PoseAssets (not synchronized with its original animation) that appears, that does not appear when running a single-process cook. I believe we can just resynchronize them in data and that will be fine - still I wonder why these appear in multi-process cooks.
Any help would be appreciated.
Thanks,
Steps to Reproduce
* Start multiple BuildCookRun -cook with “-AdditionalCookerOptions=-cookprocesscount=3” (or any other value)
* Observe that some World Partition worlds logs in <ProjectName>\Saved\Logs\WorldPartition can differ on HLODRelevant and RuntimeHLODLayer values
* Observe that some World Partition worlds logs seems duplicated on a single multi-process cook run of BuildCookRun -cook, and that the one having wrongly set values are always written some seconds or minutes after the right one; even one run had three different logs, when cooking with 3 processes.
* It seems to happen much more on very small World Partition worlds.
* False setting to true of HLODRelevant and assigning of RuntimeHLODLayer are always happening on instances of blueprints deriving from PackedLevelActor.
* Logs could be erroneous, but functionality could still be ok.
* Nothing in cook logs mention anything about HLODs, but there are many warnings that appear (especially related to PoseAsset) that don’t appear in regular single-process cooks.
Sorry to respond late to this.
Indeterminism of HLODRelevant and RuntimeHLODLayer, only when running MPCook
I don’t have any immediate ideas; I am researching possibilities for what might be happening now. Have you been able to reproduce this in a test project like Lyra? Have you been able to reproduce it in new levels that you create in your own project? Sources of indeterminism in the cook often come from which types of content are used. Using level instances for example, and using embedded level instances versus dynamic level instances; both of those have had unique non-determinism bugs of their own in the past. And those are just the cases I have seen, there may be other specific content features that cause this as well. And being specific to MPCook might be because MPCook allows can cook some generated streaming cell packages of maps in a different order, and sometimes even split them up to different cookworkers, so there may be some values that get initialized from the persistent level or from one streaming cell that are then written out in another streaming cell. How many packages are cooked in your project?
“Poses are out-of-sync with the source animation” intermittent warnings, only when running MPCook
We have seen these warnings locally, but not intermittently, and not specific to MPCook. It might be related to more garbage collection occurring during MPCook. How many occurrences of “LogCook: Display: Garbage collection triggered (Full)” do you see in the CookDirector’s log during MPCook? How many do you see in SPCook?
Other warnings that only appear when running MPCook
Increased garbage collection is a cause of multiple system-specific issues, but other than that we don’t know of any systemic warnings present in MPCook that are not present in SPCook. There are some specific warnings and bugs we have fixed though. Can you post all the warnings you have seen, that only occur during MPCook?
Hello !
Late answer, I’m on other matters right now. We have not tried to reproduce in a sample project like Lyra, and we didn’t try on sample gym levels. The number of packages is currently above 100000 right now. As for the different ordering, I would retry in SP Cook with -RANDOMPACKAGEORDER on a dozen of cooks and see what happens.
As for the poses that are out-of-sync, we will just fix them in data and we should be fine.
Otherwise, the MPcook build seems to run without any problem; I’m just wondering if the HLODRelevant/RuntimeHLODLayer could be false positives.
I haven’t been able to reproduce the HLODRelevant/RuntimeHLODLayer issue in Lyra, and I don’t see it in a few samples of larger projects from multiple cooks. So what you’re seeing is unexpected and not a false positive. It’s not necessarily a problem, though - indeterminism in the logs does not necessarily mean indeterminism in the cooked output.
I haven’t been able to find any leads through static analysis; there are some vectors it might be coming from (system-specific AActor::IsHLODRelevant functions, order of events on FContainerCollectionInstanceDescriptor::bIsHLODRelevant), but nothing that is obviously a problem.
I don’t think you need to investigate this now and you can ignore it until an unless you find indeterminism in the cooked output. If that does occur, or if you want to investigate it just because its something unexpected, for the next steps of the investigation I think you should try to find a repro case in your content - does it occur consistently enough in some assets that we can add instrumentation specific to those assets to trace back where it is coming from.
Thanks. Meanwhile, on build machines dedicated to monitor MPcook usability, we have a sporadic EDL graph error with `Error: Content is missing from cook. Source package referenced an object in target package but the object was stripped out of the target package when saved.’ that appears from a _Generated_ package, with always the same target BP package and same referenced object that I need to dig-up if there is any resave from an original WP actor that could fix this, although I will need to dig what actor could cause this.
There is a known issue with Niagara packages having indeterminate object names within them; Content Missing from Cook errors can then occur in source packages that have references to target Niagara packages. We are working on fixing that in 5.8; we unfortunately were not able to finish it in time for 5.7.0 and it is too complicated for a hotfix.
Is the target package in these cases a Niagara package?
Roger, this is a new unreported issue then, let me know if you want to collaborate on debugging it.
No, it is a game-specific class.