Intermittently we are seeing issues with Horde Artifacts hitting rate limiting on the AWS S3 API at the end of a job step in Horde.
The attached stack trace occurs after uploading a large amount of files to Horde storage -
Written 5947 files (49409.8mb, 2472.8mb/s)
Reading block "Compile Tools Win64 Development":"ToolsFiles_Win64_Development" from temp storage (artifact: 68f618cbcc5b4f736ce2ac3a 'compile-tools-win64-development' (step-output), ns: horde-artifacts, ref: step-output/REDACTED_dev-ue5/3934999/compile-tools-win64-development/68f618cbcc5b4f736ce2ac3a, local: T:\HordeData\StandardCi\Sync\Engine\Saved\BuildGraph\Compile Tools Win64 Development\Manifest-ToolsFiles_Win64_Development.xml, blockdir: block-ToolsFiles_Win64_Development)
Using 16 read tasks, 16 decode tasks, 16 write tasks
Written 0 files (0.0mb, 0.0mb/s)
Exception while executing step: EpicGames.Horde.Storage.StorageException:
...
Is there any configuration or setting to self-limit the upload rate so we don’t hit this? Otherwise should the storage backend be waiting and retrying?
As far as I am aware, there’s no account level request to increase S3 upload resources in AWS