Following up and signal boosting of a previous thread: [Content removed]
And PR: https://github.com/EpicGames/UnrealEngine/pull/13349
tl;dr: Horde’s BatchOutputWriter implementation is significantly slower at unpacking large files (>1GB), leading to disproportionate delay when such artifacts need to be transferred between agents.
I have before and after numbers (based on 5.6.0):
Before:
Reading block "Compile SparkleServer Win64":"Server Binaries Win64+Server DebugSymbols Win64" from temp storage (artifact: 6887a9fb486144eb83401c6a 'compile-sparkleserver-win64' (step-output), ns: horde-artifacts, ref: step-output/sparkle-main/69078/compile-sparkleserver-win64/6887a9fb486144eb83401c6a, local: D:\h\sparkle\Sync\Engine\Saved\BuildGraph\Compile SparkleServer Win64\Manifest-Server Binaries Win64+Server DebugSymbols Win64.xml, blockdir: block-Server Binaries Win64+Server DebugSymbols Win64) Using 16 read tasks, 16 decode tasks, 16 write tasks Written 1 files (11.1mb, 2.9mb/s) Written 1 files (28.9mb, 3.8mb/s) Written 2 files (46.6mb, 3.7mb/s) Written 3 files (65.8mb, 3.7mb/s) Written 3 files (84.3mb, 3.6mb/s) Written 3 files (102.6mb, 3.6mb/s) Written 3 files (123.1mb, 4.1mb/s) Written 3 files (144.2mb, 3.8mb/s) Written 3 files (162.1mb, 3.6mb/s) Written 3 files (184.3mb, 4.7mb/s) Written 4 files (207.0mb, 4.3mb/s) Written 4 files (229.2mb, 4.5mb/s) Written 4 files (252.3mb, 4.2mb/s) Written 5 files (277.2mb, 4.8mb/s) Written 5 files (299.2mb, 4.8mb/s) Written 5 files (321.1mb, 4.0mb/s) Written 6 files (342.3mb, 3.9mb/s) Written 6 files (358.1mb, 3.2mb/s) Written 7 files (373.9mb, 3.4mb/s) Elapsed: 96s, bundle.packet_cache.hits: 84, bundle.packet_cache.misses: 10, bundle.bundle_cache.hits: 10, bundle.bundle_cache.misses: 0, backend.http.wall_time_secs: 0, backend.http.num_bytes: 0, bundle.packet_reader.num_bytes_read: 0, bundle.packet_reader.num_encoded_bytes_read: 0, bundle.packet_reader.num_decoded_bytes_read: 0
After:
Reading block "Compile SparkleServer Win64":"Server Binaries Win64+Server DebugSymbols Win64" from temp storage (artifact: 6887b0d2486144eb834032f0 'compile-sparkleserver-win64' (step-output), ns: horde-artifacts, ref: step-output/sparkle-main/69082/compile-sparkleserver-win64/6887b0d2486144eb834032f0, local: D:\h\sparkle\Sync\Engine\Saved\BuildGraph\Compile SparkleServer Win64\Manifest-Server Binaries Win64+Server DebugSymbols Win64.xml, blockdir: block-Server Binaries Win64+Server DebugSymbols Win64) Using 16 read tasks, 16 decode tasks, 16 write tasks Written 7 files (373.9mb, 110.7mb/s) Elapsed: 3s, bundle.packet_cache.hits: 84, bundle.packet_cache.misses: 10, bundle.bundle_cache.hits: 10, bundle.bundle_cache.misses: 0, backend.http.wall_time_secs: 0, backend.http.num_bytes: 0, bundle.packet_reader.num_bytes_read: 0, bundle.packet_reader.num_encoded_bytes_read: 0, bundle.packet_reader.num_decoded_bytes_read: 0