Horde: "Upload To Horde Storage For UGS" step fails the http PUT

Development Pipeline & Plugins
, , ,
1

Hey all,

we have a dedicated server machine in the cloud running two VMs, one for the Perforce and one for the Horde Server.
I’ve gotten past 98% of the setup, config issues and other problems. But now I’m stuck.

Using a on-prem build machine as a Horde Agent I run the “Incremental Build“ job to create the Precompile Binaries for distribution. And on the very last step to “Upload to Horde Storage for UGS“ I get a PUT failed error.

I’ve attached the Log for the Horde Job. I certainly hope we can remedy this.
Any help is appreciated.

Kind Regards,

Jannis

(I’m unable to upload the .log due to being a new user.. sorry.)

Summary

Reading tag “Stage for UGS”:“#Stage for UGS” from D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Saved\BuildGraph\Stage for UGS\Tag-Stage for UGS.xml
Reading tag “Stage for UGS”:“#PublishBinaries” from D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Saved\BuildGraph\Stage for UGS\Tag-PublishBinaries.xml
Reading tag “Tag Output Files”:“#OutputFiles” from D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Saved\BuildGraph\Tag Output Files\Tag-OutputFiles.xml
Reading block “Stage for UGS”:“PublishBinaries” from D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Saved\BuildGraph\Stage for UGS\Manifest-PublishBinaries.xml
Reading block “Tag Output Files”:“OutputFiles” from D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Saved\BuildGraph\Tag Output Files\Manifest-OutputFiles.xml
Download took 0,0s
Setting env var: IsBuildMachine=1
Setting env var: P4CLIENT=Horde+DESKTOP-NAME+PROJECTNAME-Inc
Setting env var: P4USER=perforceserversuperuser
Setting env var: UE_HORDE_AVAILABILITY_ZONE=
Setting env var: UE_HORDE_BATCHID=14df
Setting env var: UE_HORDE_CLEANUP=D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Cleanup.bat
Setting env var: UE_HORDE_GRAPH_UPDATE=D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Saved\Horde\Graph.json
Setting env var: UE_HORDE_JOBID=691f7ef8600613f459b4a0d1
Setting env var: UE_HORDE_LEASE_CLEANUP=D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\CleanupLease.bat
Setting env var: UE_HORDE_LEASEID=691faecc9c1d4ffa92ff71f6
Setting env var: UE_HORDE_REGION=
Setting env var: UE_HORDE_SHARED_DIR=D:\HordeAgent\Sandbox\Saved
Setting env var: UE_HORDE_STEPID=26f1
Setting env var: UE_HORDE_STEPNAME=Submit To Horde Storage For UGS
Setting env var: UE_HORDE_STREAMID=PROJECTNAMEmain
Setting env var: UE_HORDE_TEMPLATEID=incremental-build
Setting env var: UE_HORDE_TEMPLATENAME=Incremental Build
Setting env var: UE_HORDE_TOKEN=[redacted]
Setting env var: UE_HORDE_URL=http://IP.IP.IP.IP:13340/
Setting env var: UE_LOG_JSON_TO_STDOUT=1
Setting env var: UE_TELEMETRY_DIR=D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Programs\AutomationTool\Saved\Telemetry
Setting env var: UE_TESTDATA_DIR=D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Programs\AutomationTool\Saved\TestData
Setting env var: UE-S3DataCachePath=None
Setting env var: uebp_BuildRoot_Escaped=++PROJECTNAME+PROJECTNAMEMain
Setting env var: uebp_BuildRoot_P4=//PROJECTNAME/PROJECTNAMEMain
Setting env var: uebp_CL=26
Setting env var: uebp_CLIENT=Horde+DESKTOP-NAME+PROJECTNAME-Inc
Setting env var: uebp_CLIENT_ROOT=//Horde+DESKTOP-NAME+PROJECTNAME-Inc
Setting env var: uebp_CodeCL=26
Setting env var: uebp_LOCAL_ROOT=D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync
Setting env var: uebp_LogFolder=D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Programs\AutomationTool\Saved\Logs
Setting env var: uebp_PORT=ssl:IP.IP.IP.IP:1666
Setting env var: uebp_USER=perforceserversuperuser
Executing C:\Windows\system32\cmd.exe /C ““D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Build\BatchFiles\RunUAT.bat” BuildGraph -Script=Engine/Build/Graph/Examples/BuildEditorAndTools.xml -Target=“Submit PCBs” -SingleNode=“Submit To Horde Storage For UGS” -set:UProjectPath=PROJECTNAME/PROJECTNAME.uproject -set:EditorTarget=PROJECTNAMEEditor -set:PreferredAgent=IncrementalWin64;CompileWin64;Win64 -set:ArchiveStream=//PROJECTNAME/PROJECTNAMEBinaries -NoCompile”
Running AutomationTool…
Using bundled DotNet SDK version: 8.0.300 win-x64
Starting AutomationTool…
Parsing command line: BuildGraph -Script=Engine/Build/Graph/Examples/BuildEditorAndTools.xml -Target=“Submit PCBs” -SingleNode=“Submit To Horde Storage For UGS” -set:UProjectPath=PROJECTNAME/PROJECTNAME.uproject -set:EditorTarget=PROJECTNAMEEditor -set:PreferredAgent=IncrementalWin64;CompileWin64;Win64 -set:ArchiveStream=//PROJECTNAME/PROJECTNAMEBinaries -NoCompile
Initializing script modules…
Total script module initialization time: 0,35 s.
Using C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Current\Bin\MSBuild.exe
Detected Perforce Settings:
Server: ssl:IP.IP.IP.IP:1666
User: perforceserversuperuser
Client: Horde+DESKTOP-NAME+PROJECTNAME-Inc
Branch: //PROJECTNAME/PROJECTNAMEMain
Last Change: 26
Last Code Change: 26
Executing commands…
Reading tag “Stage for UGS”:“#Stage for UGS” from local file D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Saved\BuildGraph\Stage for UGS\Tag-Stage for UGS.xml
Reading tag “Stage for UGS”:“#PublishBinaries” from local file D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Saved\BuildGraph\Stage for UGS\Tag-PublishBinaries.xml
Reading tag “Tag Output Files”:“#OutputFiles” from local file D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Saved\BuildGraph\Tag Output Files\Tag-OutputFiles.xml
Reading tag “Stage for UGS”:“PublishBinaries” manifest from D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Saved\BuildGraph\Stage for UGS\Manifest-PublishBinaries.xml
Reading tag “Tag Output Files”:“OutputFiles” manifest from D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Saved\BuildGraph\Tag Output Files\Manifest-OutputFiles.xml

========== Starting: Submit To Horde Storage For UGS ==========
Creating artifact 691fafa79c1d4ffa92ff7343 ‘editor’ (ugs-pcb) from 4125 files (5.653,5mb). Namespace: horde-artifacts, ref ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343 (http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/refs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343)
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_2 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_2 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6b537fb0d8a94043b579dbf6d89b42ca_2 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_2 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/aec4606b7fef47a5aedaa702994f3774_2 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6043d041d2554f51930249ea0ea30097_3 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/dcd1bc53ec4b438fa31b2c344e4664e1_3 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/a772a15624f04582951e2fd6f4e28083_3 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/d4767deb4d3446ddb9d1813f0836e337_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/3ec4e68759344cc9b36a8e459dca56de_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/bb2510e296654b6f89532e7f83609c7c_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/57d40c7152d14681bd3756b5b77ba345_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6b537fb0d8a94043b579dbf6d89b42ca_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/d4767deb4d3446ddb9d1813f0836e337_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/3ec4e68759344cc9b36a8e459dca56de_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/57d40c7152d14681bd3756b5b77ba345_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/bb2510e296654b6f89532e7f83609c7c_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6b537fb0d8a94043b579dbf6d89b42ca_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6043d041d2554f51930249ea0ea30097_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/aec4606b7fef47a5aedaa702994f3774_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/aec4606b7fef47a5aedaa702994f3774_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6043d041d2554f51930249ea0ea30097_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7a82f78c9cbf4f0b89995d3ba59bfc9c_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7a82f78c9cbf4f0b89995d3ba59bfc9c_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/920f465e27c94a2dafe46d075564790b_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/920f465e27c94a2dafe46d075564790b_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/a772a15624f04582951e2fd6f4e28083_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/a772a15624f04582951e2fd6f4e28083_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7a82f78c9cbf4f0b89995d3ba59bfc9c_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7a82f78c9cbf4f0b89995d3ba59bfc9c_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/3ec4e68759344cc9b36a8e459dca56de_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/57d40c7152d14681bd3756b5b77ba345_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/bb2510e296654b6f89532e7f83609c7c_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/d4767deb4d3446ddb9d1813f0836e337_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6043d041d2554f51930249ea0ea30097_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/3ec4e68759344cc9b36a8e459dca56de_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/57d40c7152d14681bd3756b5b77ba345_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/bb2510e296654b6f89532e7f83609c7c_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/d4767deb4d3446ddb9d1813f0836e337_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6043d041d2554f51930249ea0ea30097_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/920f465e27c94a2dafe46d075564790b_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/920f465e27c94a2dafe46d075564790b_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/a772a15624f04582951e2fd6f4e28083_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/a772a15624f04582951e2fd6f4e28083_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/dcd1bc53ec4b438fa31b2c344e4664e1_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/dcd1bc53ec4b438fa31b2c344e4664e1_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_3 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/bb2510e296654b6f89532e7f83609c7c_3 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_3 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/bb2510e296654b6f89532e7f83609c7c_3 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_3 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/57d40c7152d14681bd3756b5b77ba345_3 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_3 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/57d40c7152d14681bd3756b5b77ba345_3 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/d4767deb4d3446ddb9d1813f0836e337_3 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/d4767deb4d3446ddb9d1813f0836e337_3 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_3 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_3 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_3 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_3 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6b537fb0d8a94043b579dbf6d89b42ca_3 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_3 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/aec4606b7fef47a5aedaa702994f3774_3 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7a82f78c9cbf4f0b89995d3ba59bfc9c_3 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/aec4606b7fef47a5aedaa702994f3774_3 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6b537fb0d8a94043b579dbf6d89b42ca_3 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_3 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/3ec4e68759344cc9b36a8e459dca56de_3 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7a82f78c9cbf4f0b89995d3ba59bfc9c_3 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/3ec4e68759344cc9b36a8e459dca56de_3 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_3 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_3 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/57d40c7152d14681bd3756b5b77ba345_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/bb2510e296654b6f89532e7f83609c7c_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/57d40c7152d14681bd3756b5b77ba345_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/bb2510e296654b6f89532e7f83609c7c_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/d4767deb4d3446ddb9d1813f0836e337_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/d4767deb4d3446ddb9d1813f0836e337_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/3ec4e68759344cc9b36a8e459dca56de_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/3ec4e68759344cc9b36a8e459dca56de_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7a82f78c9cbf4f0b89995d3ba59bfc9c_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6b537fb0d8a94043b579dbf6d89b42ca_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7a82f78c9cbf4f0b89995d3ba59bfc9c_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6b537fb0d8a94043b579dbf6d89b42ca_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/920f465e27c94a2dafe46d075564790b_3 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/920f465e27c94a2dafe46d075564790b_3 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/dcd1bc53ec4b438fa31b2c344e4664e1_1 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/dcd1bc53ec4b438fa31b2c344e4664e1_1 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6043d041d2554f51930249ea0ea30097_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6043d041d2554f51930249ea0ea30097_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/920f465e27c94a2dafe46d075564790b_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/920f465e27c94a2dafe46d075564790b_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/a772a15624f04582951e2fd6f4e28083_3 failed (null). Delaying for 5000ms (attempt #2).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/bb2510e296654b6f89532e7f83609c7c_4 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_3 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6b537fb0d8a94043b579dbf6d89b42ca_2 failed (null). Delaying for 5000ms (attempt #2).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/aec4606b7fef47a5aedaa702994f3774_2 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/aec4606b7fef47a5aedaa702994f3774_2 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6b537fb0d8a94043b579dbf6d89b42ca_5 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_6 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_5 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/920f465e27c94a2dafe46d075564790b_5 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_6 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/aec4606b7fef47a5aedaa702994f3774_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/a772a15624f04582951e2fd6f4e28083_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/aec4606b7fef47a5aedaa702994f3774_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/a772a15624f04582951e2fd6f4e28083_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/3ec4e68759344cc9b36a8e459dca56de_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7a82f78c9cbf4f0b89995d3ba59bfc9c_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/3ec4e68759344cc9b36a8e459dca56de_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7a82f78c9cbf4f0b89995d3ba59bfc9c_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6043d041d2554f51930249ea0ea30097_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6043d041d2554f51930249ea0ea30097_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/d4767deb4d3446ddb9d1813f0836e337_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/d4767deb4d3446ddb9d1813f0836e337_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/dcd1bc53ec4b438fa31b2c344e4664e1_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/dcd1bc53ec4b438fa31b2c344e4664e1_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_4 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_4 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/dcd1bc53ec4b438fa31b2c344e4664e1_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/dcd1bc53ec4b438fa31b2c344e4664e1_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/57d40c7152d14681bd3756b5b77ba345_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/57d40c7152d14681bd3756b5b77ba345_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7a82f78c9cbf4f0b89995d3ba59bfc9c_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/bb2510e296654b6f89532e7f83609c7c_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/a772a15624f04582951e2fd6f4e28083_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/bb2510e296654b6f89532e7f83609c7c_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6043d041d2554f51930249ea0ea30097_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7a82f78c9cbf4f0b89995d3ba59bfc9c_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/a772a15624f04582951e2fd6f4e28083_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/3ec4e68759344cc9b36a8e459dca56de_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6b537fb0d8a94043b579dbf6d89b42ca_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6043d041d2554f51930249ea0ea30097_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/bb2510e296654b6f89532e7f83609c7c_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/3ec4e68759344cc9b36a8e459dca56de_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6b537fb0d8a94043b579dbf6d89b42ca_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/bb2510e296654b6f89532e7f83609c7c_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/920f465e27c94a2dafe46d075564790b_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/920f465e27c94a2dafe46d075564790b_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/57d40c7152d14681bd3756b5b77ba345_7 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/dcd1bc53ec4b438fa31b2c344e4664e1_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/57d40c7152d14681bd3756b5b77ba345_7 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/dcd1bc53ec4b438fa31b2c344e4664e1_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_7 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/bb2510e296654b6f89532e7f83609c7c_7 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_7 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/d4767deb4d3446ddb9d1813f0836e337_7 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/bb2510e296654b6f89532e7f83609c7c_7 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/d4767deb4d3446ddb9d1813f0836e337_7 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6043d041d2554f51930249ea0ea30097_7 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_7 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6043d041d2554f51930249ea0ea30097_7 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_7 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_7 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_7 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/aec4606b7fef47a5aedaa702994f3774_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/aec4606b7fef47a5aedaa702994f3774_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/d4767deb4d3446ddb9d1813f0836e337_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/d4767deb4d3446ddb9d1813f0836e337_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_7 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_7 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_7 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/920f465e27c94a2dafe46d075564790b_7 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_7 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/920f465e27c94a2dafe46d075564790b_7 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/57d40c7152d14681bd3756b5b77ba345_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/57d40c7152d14681bd3756b5b77ba345_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/aec4606b7fef47a5aedaa702994f3774_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/aec4606b7fef47a5aedaa702994f3774_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/3ec4e68759344cc9b36a8e459dca56de_7 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/3ec4e68759344cc9b36a8e459dca56de_7 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_8 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_8 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_7 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_7 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_8 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/2890cd0fbdc64bafaf310c51e094afd4_8 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_8 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/eba3b674074c499b94b6e699bb49acf8_8 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6b537fb0d8a94043b579dbf6d89b42ca_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/6b537fb0d8a94043b579dbf6d89b42ca_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/7c546f650f7d492196df03336d5b0eb2_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/920f465e27c94a2dafe46d075564790b_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_5 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/920f465e27c94a2dafe46d075564790b_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_5 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_6 timed out after 60s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1a00f8246b374f9dbc932e39d5ed2e5d_6 retrying after 5s.
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/01af30fabd224639b77d07edd54331ed_6 failed (null). Delaying for 1000ms (attempt #1).
PUT http://IP.IP.IP.IP:13340/api/v1/storage/horde-artifacts/blobs/ugs-pcb/PROJECTNAMEmain/26/editor/691fafa79c1d4ffa92ff7343/1d096577db334ac4b5bf6d3ddf47425e_5 failed (null). Delaying for 1000ms (attempt #1).
System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
—> System.TimeoutException: The operation was canceled.
—> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
—> System.Net.Http.HttpRequestException: Error while copying content to a stream.
—> System.IO.IOException: Unable to write data to the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
—> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
— End of inner exception stack trace —
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.IO.Stream.g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at System.Net.Http.HttpContent.g__WaitAsync|56_0(ValueTask copyTask)
— End of inner exception stack trace —
at System.Net.Http.HttpContent.g__WaitAsync|56_0(ValueTask copyTask)
at System.Net.Http.MultipartContent.SerializeToStreamAsyncCore(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.HttpContent.g__WaitAsync|56_0(ValueTask copyTask)
at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
— End of inner exception stack trace —
at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendCoreAsync(HttpRequestMessage request, Context context, CancellationToken cancellationToken)
at Polly.Timeout.AsyncTimeoutEngine.ImplementationAsync[TResult](Func3 action, Context context, CancellationToken cancellationToken, Func2 timeoutProvider, TimeoutStrategy timeoutStrategy, Func5 onTimeoutAsync, Boolean continueOnCapturedContext) at Polly.Timeout.AsyncTimeoutEngine.ImplementationAsync[TResult](Func3 action, Context context, CancellationToken cancellationToken, Func2 timeoutProvider, TimeoutStrategy timeoutStrategy, Func5 onTimeoutAsync, Boolean continueOnCapturedContext)
at Polly.AsyncPolicy1.ExecuteAsync(Func3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
at Polly.Wrap.AsyncPolicyWrapEngine.<>c__DisplayClass2_01.<<ImplementationAsync>b__0>d.MoveNext() --- End of stack trace from previous location --- at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates1 shouldRetryResultPredicates, Func5 onRetryAsync, Int32 permittedRetryCount, IEnumerable1 sleepDurationsEnumerable, Func4 sleepDurationProvider, Boolean continueOnCapturedContext)
at Polly.AsyncPolicy.ExecuteAsync[TResult](Func3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext) at Polly.Wrap.AsyncPolicyWrapEngine.ImplementationAsync[TResult](Func3 func, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext, IAsyncPolicy outerPolicy, IAsyncPolicy1 innerPolicy) at Polly.AsyncPolicy1.ExecuteAsync(Func3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext) at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendCoreAsync(HttpRequestMessage request, Context context, CancellationToken cancellationToken) at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates1 shouldRetryResultPredicates, Func5 onRetryAsync, Int32 permittedRetryCount, IEnumerable1 sleepDurationsEnumerable, Func4 sleepDurationProvider, Boolean continueOnCapturedContext)
at Polly.AsyncPolicy1.ExecuteAsync(Func3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
— End of inner exception stack trace —
— End of inner exception stack trace —
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at EpicGames.Horde.Storage.Backends.HttpStorageBackend.SendWriteRequestAsync(Nullable1 locator, StreamContent streamContent, IReadOnlyCollection1 imports, String prefix, CancellationToken cancellationToken) in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Backends\HttpStorageBackend.cs:line 213
at EpicGames.Horde.Storage.Backends.HttpStorageBackend.WriteBlobAsync(Nullable1 locator, Stream stream, IReadOnlyCollection1 imports, String prefix, CancellationToken cancellationToken) in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Backends\HttpStorageBackend.cs:line 178
at EpicGames.Horde.Storage.Bundles.V2.BundleWriter.PendingBundleHandle.WriteAsync(CancellationToken cancellationToken) in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Bundles\V2\BundleWriter.cs:line 222
at EpicGames.Horde.Storage.Bundles.V2.BundleWriter.HandleBundleWriteAsync(Task lockTask, PendingBundleHandle bundle) in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Bundles\V2\BundleWriter.cs:line 376
at EpicGames.Horde.Storage.Bundles.V2.BundleWriter.FlushAsync(CancellationToken cancellationToken) in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Bundles\V2\BundleWriter.cs:line 406
at EpicGames.Horde.Storage.Bundles.V2.BundleWriter.DisposeAsync() in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Bundles\V2\BundleWriter.cs:line 332
at EpicGames.Horde.Storage.Nodes.DirectoryNodeUpdateExtensions.CreateLeafChunkNodesAsync(IBlobWriter writer, IReadOnlyList1 files, LeafChunkedData[] leafChunks, Int32 start, Int32 count, UpdateStats updateStats, ChunkingOptions options, CancellationToken cancellationToken) in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Nodes\DirectoryNode.Update.cs:line 567 at System.Threading.Tasks.Parallel.<>c__531.<b__53_0>d.MoveNext()
— End of stack trace from previous location —
at EpicGames.Horde.Storage.Nodes.DirectoryNodeUpdateExtensions.AddFilesAsync(DirectoryNode directoryNode, DirectoryReference baseDir, IEnumerable1 files, IBlobWriter writer, ChunkingOptions options, IProgress1 progress, CancellationToken cancellationToken) in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Nodes\DirectoryNode.Update.cs:line 499
at EpicGames.Horde.Storage.Nodes.DirectoryNodeUpdateExtensions.AddFilesAsync(IBlobWriter writer, DirectoryReference baseDir, IReadOnlyList1 files, ChunkingOptions options, IProgress1 progress, CancellationToken cancellationToken) in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Nodes\DirectoryNode.Update.cs:line 377
at AutomationTool.Tasks.CreateArtifactTask.ExecuteAsync(JobContext job, HashSet1 buildProducts, Dictionary2 tagNameToFileSet) in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\AutomationTool\BuildGraph\Tasks\CreateArtifactTask.cs:line 246
at AutomationTool.Tasks.CreateArtifactTask.ExecuteAsync(JobContext job, HashSet1 buildProducts, Dictionary2 tagNameToFileSet) in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\AutomationTool\BuildGraph\Tasks\CreateArtifactTask.cs:line 246
at AutomationTool.BgScriptNodeExecutor.Execute(JobContext job, Dictionary2 tagNameToFileSet) in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\AutomationTool\BuildGraph\BgNodeExecutor.cs:line 378 at AutomationTool.BuildGraph.BuildNodeAsync(BgGraphDef graph, BgNodeDef node, Dictionary2 nodeToExecutor, TempStorage storage, Boolean withBanner) in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\AutomationTool\BuildGraph\BuildGraph.cs:line 1138
at AutomationTool.BuildGraph.ExecuteAsync() in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\AutomationTool\BuildGraph\BuildGraph.cs:line 748
at AutomationTool.Automation.ExecuteAsync(List1 CommandsToExecute, Dictionary2 Commands) in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\AutomationTool\AutomationUtils\Automation.cs:line 284
at AutomationTool.Automation.ExecuteAsync(List1 CommandsToExecute, Dictionary2 Commands) in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\AutomationTool\AutomationUtils\Automation.cs:line 308
at AutomationTool.Automation.ProcessAsync(ParsedCommandLine AutomationToolCommandLine, StartupTraceListener StartupListener, HashSet`1 ScriptModuleAssemblies) in D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Source\Programs\AutomationTool\AutomationUtils\Automation.cs:line 168
while executing task
at D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Build\Graph\Examples\BuildEditorAndTools.xml(209)
(see D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Programs\AutomationTool\Saved\Logs\Log.txt for full exception trace)
AutomationTool executed for 0h 3m 24s
AutomationTool exiting with ExitCode=1 (Error_Unknown)
BUILD FAILED
Reading telemetry from D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Programs\AutomationTool\Saved\Telemetry\AutomationTool.dll.26976.638992846610145707.json
Reading telemetry from D:\HordeAgent\Sandbox\PROJECTNAME-Inc\Sync\Engine\Programs\AutomationTool\Saved\Telemetry\AutomationTool.dll.noscope.26976.opentracing.json
Creating artifact 691fb0729c1d4ffa92ff7597 ‘submit-to-horde-storage-for-ugs’ (step-trace) from 3 files (0,0mb). Namespace horde-artifacts, ref step-trace/PROJECTNAMEmain/26/submit-to-horde-storage-for-ugs/691fb0729c1d4ffa92ff7597 (http://IP.IP.IP.IP:13340//api/v1/storage/horde-artifacts/refs/step-trace/PROJECTNAMEmain/26/submit-to-horde-storage-for-ugs/691fb0729c1d4ffa92ff7597)
Uploaded artifact 691fb0729c1d4ffa92ff7597 in 0,1s (0,1mb/s, 0,6mbps)
Creating artifact 691fb0739c1d4ffa92ff75a1 ‘submit-to-horde-storage-for-ugs’ (step-saved) from 2 files (0,2mb). Namespace horde-artifacts, ref step-saved/PROJECTNAMEmain/26/submit-to-horde-storage-for-ugs/691fb0739c1d4ffa92ff75a1 (http://IP.IP.IP.IP:13340//api/v1/storage/horde-artifacts/refs/step-saved/PROJECTNAMEmain/26/submit-to-horde-storage-for-ugs/691fb0739c1d4ffa92ff75a1)
Uploaded artifact 691fb0739c1d4ffa92ff75a1 in 0,1s (1,4mb/s, 11,1mbps)

Update:
After continued testing, I have a strong feeling it’s due to the slow upload speed that I have from my remote setup (including the Horde Agent build machine). So first order of business, what server and/or client settings could i change to allow for very slow uploads?

2

Update:

I managed to circumvent the issue by using a considerably faster internet connection.
From 20mb/s → 250mb/s upload.

This doesn’t answer or resolve my query though. We need Horde Server and Agents to be able to support slow networks.

I would greatly appreciate some clarification on what configs could be changed and where to increase “upload & download timeouts“!
And I believe this could greatly increase the availability of these tools to countries and people without gigabit connections.