Looking at the agent logs was the ticket, I’m seeing this when trying to upload a log blob to storage:
2025-05-06 11:46:39.194 -07:00 [WRN] Unable to flush data to server (log 681a55c08456fa71c979fc9c, offset 40431) EpicGames.Horde.Storage.StorageException: Unable to upload data to redirected URL <Secure URL removed>. Status:BadRequest Reason:An HTTP header that's mandatory for this request is not specified. Response:<?xml version="1.0" encoding="utf-8"?> <Error><Code>MissingRequiredHeader</Code><Message>An HTTP header that's mandatory for this request is not specified. RequestId:0eb27bb7-a01e-0001-5bb7-bef661000000 Time:2025-05-06T18:46:39.1753500Z</Message><HeaderName>x-ms-blob-type</HeaderName></Error> at EpicGames.Horde.Storage.Backends.HttpStorageBackend.WriteBlobAsync(Nullable
1 locator, Stream stream, IReadOnlyCollection1 imports, String prefix, CancellationToken cancellationToken) in D:\P4\Epic\Release-5.5\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Backends\HttpStorageBackend.cs:line 162 at EpicGames.Horde.Storage.Bundles.V2.BundleWriter.PendingBundleHandle.FlushAsync(CancellationToken cancellationToken) in D:\P4\Epic\Release-5.5\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Bundles\V2\BundleWriter.cs:line 214 at EpicGames.Horde.Storage.Bundles.V2.BundleWriter.FlushAsync(CancellationToken cancellationToken) in D:\P4\Epic\Release-5.5\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Bundles\V2\BundleWriter.cs:line 314 at EpicGames.Horde.Logs.LogBuilder.FlushAsync(IBlobWriter writer, Boolean complete, CancellationToken cancellationToken) in D:\P4\Epic\Release-5.5\Engine\Source\Programs\Shared\EpicGames.Horde\Logs\LogNode.cs:line 300 at EpicGames.Horde.Logs.ServerLogger.WriteOutputAsync(ReadOnlyMemory
1 data, Boolean flush, CancellationToken cancellationToken) in D:\P4\Epic\Release-5.5\Engine\Source\Programs\Shared\EpicGames.Horde\Logs\ServerLogger.cs:line 351
at EpicGames.Horde.Logs.ServerLogger.RunDataWriterAsync() in D:\P4\Epic\Release-5.5\Engine\Source\Programs\Shared\EpicGames.Horde\Logs\ServerLogger.cs:line 223
2025-05-06 11:46:39.196 -07:00 [INF] Disposing json log task
2025-05-06 11:46:39.196 -07:00 [INF] Cancelling long poll from client side (complete)
2025-05-06 11:46:39.237 -07:00 [INF] Log 681a55c08456fa71c979fc9c tail next = -1
2025-05-06 11:46:39.237 -07:00 [INF] Finishing log tail task
2025-05-06 11:46:39.349 -07:00 [ERR] Unhandled exception while executing lease 681a55c08456fa71c979fc9b: Unable to upload data to redirected URL . Status:BadRequest Reason:An HTTP header that’s mandatory for this request is not specified. Response:<?xml version="1.0" encoding="utf-8"?>
MissingRequiredHeader
An HTTP header that's mandatory for this request is not specified.
RequestId:0eb27c49-a01e-0001-63b7-bef661000000
Time:2025-05-06T18:46:39.3296731Zx-ms-blob-type
EpicGames.Horde.Storage.StorageException: Unable to upload data to redirected URL . Status:BadRequest Reason:An HTTP header that’s mandatory for this request is not specified. Response:<?xml version="1.0" encoding="utf-8"?>
MissingRequiredHeader
An HTTP header that's mandatory for this request is not specified.
RequestId:0eb27c49-a01e-0001-63b7-bef661000000
Time:2025-05-06T18:46:39.3296731Zx-ms-blob-type
at EpicGames.Horde.Storage.Backends.HttpStorageBackend.WriteBlobAsync(Nullable1 locator, Stream stream, IReadOnlyCollection
1 imports, String prefix, CancellationToken cancellationToken) in D:\P4\Epic\Release-5.5\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Backends\HttpStorageBackend.cs:line 162
at EpicGames.Horde.Storage.Bundles.V2.BundleWriter.PendingBundleHandle.FlushAsync(CancellationToken cancellationToken) in D:\P4\Epic\Release-5.5\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Bundles\V2\BundleWriter.cs:line 214
at EpicGames.Horde.Storage.Bundles.V2.BundleWriter.FlushAsync(CancellationToken cancellationToken) in D:\P4\Epic\Release-5.5\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Bundles\V2\BundleWriter.cs:line 314
at EpicGames.Horde.Storage.Bundles.V2.BundleWriter.DisposeAsync() in D:\P4\Epic\Release-5.5\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Bundles\V2\BundleWriter.cs:line 307
at EpicGames.Horde.Logs.ServerLogger.DisposeAsync() in D:\P4\Epic\Release-5.5\Engine\Source\Programs\Shared\EpicGames.Horde\Logs\ServerLogger.cs:line 123
at HordeAgent.Leases.Handlers.ConformHandler.ExecuteAsync(ISession session, LeaseId leaseId, ConformTask conformTask, Tracer tracer, ILogger localLogger, CancellationToken cancellationToken) in D:\P4\Epic\Release-5.5\Engine\Source\Programs\Horde\HordeAgent\Leases\Handlers\ConformHandler.cs:line 46
at HordeAgent.Leases.LeaseHandler.HandleLeasePayloadAsync(ISession session, Tracer tracer, LeaseLoggerFactory leaseLoggerFactory) in D:\P4\Epic\Release-5.5\Engine\Source\Programs\Horde\HordeAgent\Leases\LeaseHandler.cs:line 167
at HordeAgent.Leases.LeaseHandler.HandleLeaseAsync(ISession session, Tracer tracer, ILogger logger, LeaseLoggerFactory leaseLoggerFactory) in D:\P4\Epic\Release-5.5\Engine\Source\Programs\Horde\HordeAgent\Leases\LeaseHandler.cs:line 118
2025-05-06 11:46:39.351 -07:00 [INF] Transitioning lease 681a55c08456fa71c979fc9b to “Completed”, outcome=“Failed”` Also to answer your question about local storage, that backend seems to work completely fine so it’s likely this missing http header error