Horde build failures with unaccessible UBA cache service

Development Pipeline & Plugins
, ,
1

Hi,

we are testing Horde-managed UBA cache in our UE5 source build and ran into a behavior we wanted to clarify.

Our setup uses the Horde UBA cache provider. UBT requests a cache session from Horde via:

POST /api/v2/compute/{clusterId}/uba-cache

Horde then calls the UBA cache service HTTP management endpoint, roughly:

http://<cacheHttpEndpoint>/addsession?..

If the UbaCacheService is unavailable, or the configured HTTP management endpoint returns an error (for example 404/connection refused/timeout), the exception currently propagates back to UBT as a ComputeClientException and fails the whole build during UBA executor initialization.

Example callstack:

UBAHordeSession.RequestCacheServer()

ServerComputeClient.AllocateUbaCacheServerAsync()

ComputeService.AllocateUbaCacheServerAsync()

HttpResponseMessage.EnsureSuccessStatusCode()

-> ComputeServiceException: Unexpected error during UBA cache server allocation

We would like to understand the intended design here.

Was it intentional that a failure to allocate a UBA cache session should fail the whole build? Or is the UBA cache expected to be treated as an optional performance optimization, where the build should continue without cache if the cache service is unavailable?

From our perspective, the latter behavior would be preferable for production CI: if the cache is unavailable, builds should still compile locally/remotely without cache, just slower. Failing the build because the optional cache service is down makes the cache a critical dependency.

As an experiment, we added a cluster-level config option, conceptually:

plugins.compute.clusters[].uba.cacheFailOpen = true

The behavior we implemented is:

- Keep the existing validation behavior for missing UBA cache config.

- Keep the existing authorization behavior for missing UbaCacheRead/UbaCacheWrite ACL.

- During UBA cache session allocation, if an exception occurs while generating/registering the cache session or calling the cache service management endpoint:

- log a warning on the Horde server,

- return an empty/no-cache UBA resource,

- let the build continue without attaching UBA cache config to the compute resource.

- Do not swallow actual request/build cancellation.

In simplified form, the server-side idea is:

try

{

// existing UBA cache session allocation

// generate session key

// call http://<cacheHttpEndpoint>/addsession

// return UbaComputeResource(cacheEndpoint, sessionKey, writeAccess)

}

catch (TaskCanceledException ex)

{

if (cacheFailOpen && !cancellationToken.IsCancellationRequested)

{

 LogWarning(...);

 return empty/no\-cache UBA resource;

}

throw;

}

catch (Exception ex)

{

if (cacheFailOpen)

{

 LogWarning(...);

 return empty/no\-cache UBA resource;

}

throw;

}

At the compute allocation call site, if the returned UBA resource has an empty cache endpoint, we do not attach it to the compute resource, so the client proceeds without UBA cache.

Does this approach match the intended architecture of Horde-managed UBA cache? Would you recommend handling this fail-open behavior in Horde server, in UBT/UBAHordeSession client code, or somewhere else?

Thanks!

Lubos Suk

[Attachment Removed]

2

Hello! This definitely looks like a bug, as we wouldn’t classify a failing cache request as a valid reason to fail a build.

I’ve raised this with the team for further investigation.

In the interim, you could try adding the fix to the UBAHordeSession.RequestCacheServer(), extending the existing try-catch block.

Let me know if that approach works out for you.

try
{
    ubaConfig = _client.AllocateUbaCacheServerAsync(clusterId, cancellationToken).GetAwaiter().GetResult();
}
catch (OperationCanceledException)
{
    return null;
}
catch (Exception ex) when (!cancellationToken.IsCancellationRequested)
{
    _logger.LogWarning(ex, "Failed to allocate UBA cache server; continuing without cache.");
    return null;
}

[Attachment Removed]

3

Hi,

Thanks for confirming this is a bug and for raising it with the team.

We actually have a similar fix already in place on our side, and it’s working well — the only difference is that instead of returning null, we return an empty UBA resource. That approach has resolved the issue for us without any side effects, so it might be worth considering for the official fix as well.

Happy to share more details on our implementation if that’s helpful.

Thanks again for looking into this.

Best regards,

Lubos

[Attachment Removed]