Recently we have noticed that various shaders, materials, and textures are being cooked non-deterministically across out nightly builds. The files that are appearing in the diff’s generated with “UneralPak iostore …” are typically .uasset’s of either shaders, materials, or textures, but sometimes .ubulk files appear as well.
I would appreciate if someone could advise on what might be going on, and what we can do to avoid this.
* Each package is built on a fresh VM using Cloud DDC (no shared), and so I’m guessing that some sort of intermediate file cache is different across builds.
* The Image we use is a clean windows environment that we prepare by installing UE5 and its prereqs, and then running a cook for all our target platforms, with the latest version of the game and engine from our project’s Perforce. We update the image once every two weeks.
We haven’t run across any material/shader determinism issues and any time those do pop up we fix them (no recent fixes though). Since the non-determinism is effecting multiple asset types maybe something systematic is going wrong specific to your cook setup.
Can you please run a test to figure out of the indeterminism reproduces with a diff only cook?
Run the cook commandlet twice with no changes, and the second time with -diffonly:
Please let me know the results of this test and we can figured out what to do next from there. If they can’t reproduce it with the steps listed above then could you please copy the output of the first cook to a different machine and run the second step there to rule out any issues with the fresh VM.
Apologies for the delay in getting back to you on this. I had to go through multiple different iterations of the test but I wasn’t able to reproduce the issue, even on two separate VM’s with the cooked folder copied to the second machine after the cook on the first.
To summarise, I think that the texture/material/shader diff I was looking at when I made the original post was more of a red herring than anything else. It looks like the main issue is that the order of the assets in the ucas file is seemingly random when built on separate machines.
If I do a diff of the two ucas files using [UnrealPak.exe iostore -list=“<ucas file path>” -csv=“output.csv”], I can see that at a certain point the order of the files starts to mutate, and every single file after that ends up either shifted up or down by the previous changes, or just moves to a completely different position in the archive. If you want to see these files please let me know and I’ll change this question to a confidential one so that I can share them with you.
I understand that we should be using a file open order config file to make this more consistent, thought we aren’t doing that yet. Without it is it safe to assume that the order of the files in the pack is random? Or is there some sort of intermediate file(s) that are produced during cook/build and cached between builds, that don’t exist on the VM when we image them?
Sorry about the late response on my part. I haven’t forgot about the issue, but I’m still in the process of getting an answer to your most recent questions regarding the pack order (outside the domain of shaders/materials).