Cooked snapshots pulled from shared zenserver appear to be incomplete

Development Pipeline & Plugins
, ,
1

When cooked output is cooked to local zen storage I am able to stream to consoles, stage with or without pak and pretty much operate normally on that snapshot. If I take that same snapshot and export it to a shared zenserver then re-import that snapshot cleanly into my local zen storage again attempt to use it there are things apparently missing. I first found this issue in our project but am now testing against the LyraStarterGame.

When streaming or even running the win64 game locally after pulling down the snapshot the behaviors vary a little but they all point to possibly things missing. An iterative or incremental cook generally ends up as a full cook and after that then the behaviors are “fixed”.

I am fairly certain this has worked for me at some point but bisecting the issue back to UE 5.6 (we are currently on 5.6.1) the issue exists in both. I have not yet attempted to bisect to 5.5.

I will also be bisecting the shared server version. Our shared zenserver is currently 5.7.4.

I am also planning on testing this against cloud-ddc to see if the results are similar.

It is possible I have missed something in the export and import commands but I believe those are similar or the same as the ones that are generated when using BuildGraph tasks to do the same actions.

UPDATE:

I have found that exporting a snapshot from local zenserver to a “shared” cloud-ddc server, the importing it back locally clean IS working. I hosted the cloud-ddc using the docker-compose from engine version 5.6.1 and ran these commands:

I forgot to mention before but the local zen.exe and zenserver.exe being used for testing are both 5.6.6. (shared zenserver is 5.7.4).

Engine\Binaries\Win64\zen.exe oplog-export --embedloosefiles --cloud http://localhost:8080 --namespace Lyra --bucket lyra --hosturl http://localhost:8558 --key 6b8d93c00e80c579d5cc6bd07216b05d054713fd --openid-provider Okta LyraStarterGame.d8744b42 Win64

Engine\Binaries\Win64\zen.exe oplog-import --clean --cloud http://localhost:8080 --namespace Lyra --bucket lyra --hosturl http://localhost:8558 --key 6b8d93c00e80c579d5cc6bd07216b05d054713fd --openid-provider Okta LyraStarterGame.d8744b42 Win64

I will be bisecting some versions for the shared zenserver instance for this shortly and will add any new info when I get it.

2

Steps to Reproduce
Repro:

  • Cook locally with cook output to zen enabled
  • Export cooked snapshot to a shared zen server
    • Engine\Binaries\Win64\zen.exe oplog-export --embedloosefiles --zen <sharedzenserver> --hosturl http://localhost:8558 --target-project lyra.oplog --target-oplog 1.win64 LyraStarterGame.d8744b42 Win64
  • Remove local snapshot from local zenserver
  • Import snapshot from shared zen server to local
    • Engine\Binaries\Win64\zen.exe oplog-import LyraStarterGame.d8744b42 Win64 --clean --hosturl localhost:8558 --zen <sharedzenserver> --source-project lyra.oplog --source-oplog 1.win64
  • Attempt to stage with without cooking and with -pak
3

Hi,

Thanks for taking time to report this. We have in fact discovered a bug introduced in recent zenserver revisions that would cause “missing chunk” issues, and I think that’s what you’re encountering. These issues were introduced around 5.7.1 and are fixed as of 5.7.6. Specifically you can see the bottom three line items in the changelog for 5.7.4, 5.7.5, and 5.7.6 all are bugfixes that fix bugs that can lead to missing data. Please let me know if these newer revisions address the issue you’re seeing.

5.7.6
Improvement: Oidc token executable is now launched hidden when launch from inside zenserver (oplog import/export)
Improvement: Helper scripts get_ue_toolchain.h and ue_build.sh can now be used without passing an argument to them. If you don't specify a toolchain directory then the default of .tmp-ue-toolchain is used
Bugfix: Fixed bug in block store GC where a chunk could be lost due to writing blocks larger than max block size
5.7.5
Bugfix: Parsing of zen oplog-import --oidctoken-exe-path option fixed
Bugfix: Fixed race condition when checking cas store block size causing chunks to be incorrectly reported as missing
5.7.4
Bugfix: Parsing of zen builds --log-progress option fixed
Bugfix: Rebuild State phase of zen builds download could read out of bound for target path array
Bugfix: Don't include oplogs pending delete when scanning for oplogs
Bugfix: Fixed issue where large ops in oplog would fail to read properly