Horde UAT Token Expired After 24h - BuildConfiguration.xml

Hello everyone,

We’ve configured Horde Server with OIDC in Azure. Authentication works on the web dashboard and with the Unreal Tool Box.

However, when I use UAT, I provide a token in my BuildConfiguration.xml file, but that token only lasts 24 hours. After that, it’s no longer usable in UAT and I have to renew it every day.

I’m using the token from: /api/v1/admin/token

Example of the BuildConfiguration.xml file:

`<?xml version="1.0" encoding="utf-8" ?>

xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance

true true false https://.com:13341 BuildMachine false false true

`

When I try to run UAT in the workspace I’m working in (for example: ./RunUAT.bat BuildCookRun -build -config=Development …)

I get the following error message because my token is more than 24 hours old:

[Image Removed]

If I use a fresh token for the day, I don’t get any error logs:

[Image Removed]

  • I’d like to know how to obtain a long-term token so I don’t have to update the BuildConfiguration.xml file every time I run UAT
  • Is it normal that the token is only valid for 24 hours ?

Hey there Maxime,

Thanks for the question. So I *think* you should be able to sidestep the token definition altogether, and have it issued via session auth from the Horde server itself based off of this change. If you try removing the <token> field from your buildconfiguration.xml, a whole other code path should be invoked. This is a mix of auto-issued HordeServer tokens (via the UE_HORDE_TOKEN environment variable - for the build automation context), or an interactive (i.e. opens browser and initiates oidc auth).

Let me know how that fares.

Kind regards,

Julian

Hi Julian,

I removed the <token> field from the buildconfiguration.xml file and applied the changes from the commit, but I’m still getting the same error message.

[Image Removed]When you mention the `UE_HORDE_TOKEN` variable—should that be something I set on my end, or is it already handled by the application?

I didn’t find that value in the documentation : https://dev.epicgames.com/documentation/en\-us/unreal\-engine/horde\-settings\-for\-unreal\-engine\#serversettings

If it helps, here are the blocks being called that correspond to the first screenshot.

ServerComputeClient.cs :[Image Removed]UBAAgentCoordinatorHorde.cs:

[Image Removed]UBAAgentCoordinatorHorde.cs :

[Image Removed]

Maxime

In case it helps, an answer to one of your questions:

“I’d like to know how to obtain a long-term token so I don’t have to update the BuildConfiguration.xml file every time I run UAT”

Is to set

"jwtExpiryTimeHours": large number,in your horde’s server.json.

This is obviously not secure nor best practice, but should work. We managed to get the ‘alternate code path’ Julian mentioned working when the token is omitted, however we had to jump through many hoops to get that to work with our auth provider (AWS cognito).

This derails the above discussion a bit, but I feel it’s worthwhile to document:

Namely, the offline_access scope is problematic as cognito does not recognize it (they handle refresh differently, I think). Fixing this involved:

  1. Proxying the cognito auth url to our own nginx server to rewrite/remove offline_access from the scopes
  2. Proxying the /api/v1/server/auth endpoint in nginx to lie about the server_url. The server_url needs to be the original cognito url in horde’s server.json, otherwise horde (in browser) complains about issuer mismatch, but it needs to be our proxy endpoint for clients (UBA/toolbox etc.) or *they* complain about issuer mismatch. The catch 22 is what necessitates intercepting this endpoint to have it both ways.

Nonetheless, while this allows everyone to communicate with cognito it breaks the refresh functionality of UBA which requires users to sign in *every* time they initiate a build. For this reason, we’re stuck using tokens as you are.

I think I saw a commit in 5.6 that would expose the scopes in the /api/v1/server/auth endpoint, which should hopefully fix this issue :slight_smile:

(Also: it was my understanding that the UE_HORDE_TOKEN was inserted for *Horde Agents* performing a build job, distinct from say a client/IDE without a horde agent installed running UBA. The separation of responsibilities and environment for agents/user clients participating in UBA and agents running ‘Horde jobs’ is a bit muddy to me.. i’m unsure if this is correct.)

Hey Maxime (and Will!),

So “UE_HORDE_TOKEN” will be utilized within the context of a build machine, within the context of a Horde Agent as the runtime wrapper. Now, the aforementioned jwyExpirtTimeHours is indeed used by the “UE_HORDE_TOKEN” code path, as well as anyone requesting a token via the /api/v1/admin/token rest endpoint.

>it was my understanding that the UE_HORDE_TOKEN was inserted for *Horde Agents* performing a build job, distinct from say a client/IDE without a horde agent installed running UBA

  • This is correct. I can see where my initial message was perhaps a bit confusing (I’ve since edited it for clarity)!

Apologies in advance for some verbosity, - I want to document some of the control flow here for verbosity. Just thinking aloud (still getting familiarized with these other use cases of OIDC), I’m curious why the interactive path is not being executed on earlier, and instead, culminating in a stale token so late. I’d be curious to get your debug log output to get a sense of what’s going on here, but below there is a bit of my investigation.

The callstack that I’m investigating is this:

> EpicGames.Horde.dll!EpicGames.Horde.HordeHttpAuthHandlerState.GetAccessTokenAsync(bool interactive, System.Threading.CancellationToken cancellationToken) Line 294 C# EpicGames.Horde.dll!EpicGames.Horde.HordeHttpAuthHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Line 63 C# System.Net.Http.dll!System.Net.Http.HttpClient.SendAsync.__Core|83_0(System.Net.Http.HttpRequestMessage request, System.Net.Http.HttpCompletionOption completionOption, System.Threading.CancellationTokenSource cts, bool disposeCts, System.Threading.CancellationTokenSource pendingRequestsCts, System.Threading.CancellationToken originalCancellationToken) Line 530 C# System.Net.Http.dll!System.Net.Http.HttpClient.SendAsync(System.Net.Http.HttpRequestMessage request, System.Net.Http.HttpCompletionOption completionOption, System.Threading.CancellationToken cancellationToken) Line 518 C# System.Net.Http.Json.dll!System.Net.Http.Json.HttpClientJsonExtensions.FromJsonAsyncCore<EpicGames.Horde.Server.GetServerInfoResponse, System.Text.Json.JsonSerializerOptions>(System.Func<System.Net.Http.HttpClient, System.Uri, System.Threading.CancellationToken, System.Threading.Tasks.Task<System.Net.Http.HttpResponseMessage>> getMethod, System.Net.Http.HttpClient client, System.Uri requestUri, System.Func<System.IO.Stream, System.Text.Json.JsonSerializerOptions, System.Threading.CancellationToken, System.Threading.Tasks.ValueTask<EpicGames.Horde.Server.GetServerInfoResponse>> deserializeMethod, System.Text.Json.JsonSerializerOptions jsonOptions, System.Threading.CancellationToken cancellationToken) Line 61 C# System.Net.Http.Json.dll!System.Net.Http.Json.HttpClientJsonExtensions.FromJsonAsyncCore<EpicGames.Horde.Server.GetServerInfoResponse>(System.Func<System.Net.Http.HttpClient, System.Uri, System.Threading.CancellationToken, System.Threading.Tasks.Task<System.Net.Http.HttpResponseMessage>> getMethod, System.Net.Http.HttpClient client, System.Uri requestUri, System.Text.Json.JsonSerializerOptions options, System.Threading.CancellationToken cancellationToken) Line 25 C# System.Net.Http.Json.dll!System.Net.Http.Json.HttpClientJsonExtensions.GetFromJsonAsync<EpicGames.Horde.Server.GetServerInfoResponse>(System.Net.Http.HttpClient client, System.Uri requestUri, System.Text.Json.JsonSerializerOptions options, System.Threading.CancellationToken cancellationToken) Line 39 C#With the following being a dispatched background task:

This is before the AddWorkerAsync invocation, and there is a lot going on in here with respect to the WindowsTokenStore (which could contain a refresh token), and acquiring the “session” token. Furthermore, placing a breakpoint inside of the OidcTokenClient constructor gives us the opportunity to track the token store, and CreateTokenManager to see how the providers are passed in.

Just to revisit this statement, and contrast it with my workflow:

  • “I’d like to know how to obtain a long-term token so I don’t have to update the BuildConfiguration.xml file every time I run UAT”
  • Mine: I occasionally get an interactive auth window that requires me to re-auth with our Horde server; which I receive a refreshed token

Just highlighting some of the most relevant code to the above flow:

`// From HordeHttpAuthHandler.cs - GetAuthStateInternalAsync
OidcTokenInfo? result = null;
if (oidcTokenManager.GetStatusForProvider(oidcProvider) != OidcStatus.NotLoggedIn)
{
try
{
result = await oidcTokenManager.TryGetAccessToken(oidcProvider, cancellationToken);
}
catch (Exception ex)
{
_logger.LogTrace(ex, “Unable to get access token; attempting login: {Message}”, ex.Message);
}
}
if (result == null && interactive)
{
_logger.LogInformation(“Logging in to {Server}…”, _serverUrl);
result = await oidcTokenManager.LoginAsync(oidcProvider, cancellationToken);
}

return new AuthState(_clock, authConfig.Method, result, interactive);`* OidcTokenManager.TryDoRefreshTokenAsync is invoked by HordeHttpAuthHandler.TryGetAccessToken
+ I’d expect your token to get refreshed here if it goes down this path
+ Just reading the code, it looks like we are enacting the _redirectUris

This has me wondering if for your azure provider flow, refreshing is just not working as expected? I would expect the refresh to fail, and then fallback to an interactive login. What happens when you blow away your oidcTokenStore.dat from C:\Users\…\AppData\Local\UnrealEngine\Common\OidcToken?

Kind regards,

Julian

Hi Julian and Will,

Julian, unfortunately the changes didn’t work, we’re seeing a lot of unexpected behavior when launching a build.

Here’s a screenshot of the error message :

[Image Removed]

However, Will’s solution setting `jwtExpiryTimeHours` is working on our end. It’s been 48 hours and the token is still valid. So we’ll continue using that approach while we wait for version 5.6.

Thank you both for your help, it allowed us to move forward. We can close the ticket.

Maxime

Hey Maxime,

Just for posterity, what change did you try there? It looks like you’re hitting a bunch of compile errors. My above suggestion was in an effort to see what occurs when you remove the oidctokenstore.dat - and force a re-auth. You should still be prompted (from a local user perspective) to perform an authentication, which should prevent any <Token> attribute being required.

Kind regards,

Julian

Hi Julian,

The change I applied was adding `“jwtExpiryTimeHours”: “87600”,` in the `server.json` file.

As for the `oidctokenstore.dat` file, I didn’t delete it because it kept refreshing every time I authenticated, so I knew it was never the same file each time.

Maxime

For what it’s worth, a different code path should be executed when the oidctokenstore.dat is entirely blown away. In either case, it does look like things are working on your end if I am not mistaken? As I mentioned above, errors you’ve posted above appear to be compile related to the Horde source - so I’m not sure what code has changed on your end.

Julian

Each time, I rolled back the changes because I wanted to return to an initial state. So I only kept the change to `jwtExpiryTimeHours`.

Yes, everything is working fine on my end. For now, I won’t touch anything else since it’s gone to production. We’ll wait for version 5.6 to try again without the token.

Thanks to you and Will for your help !