We are running a build job on a CI machine which is logged in as the standard windows service account. When it goes to authenticate with the horde server it fails to create the browser process which would allow it to authenticate:
Unable to create Horde session: An error occurred trying to start process <url with challenge> with working directory ‘D:\bld\sat\UnrealEngine\Engine\Source’. The operation attempted is not supported.
This works fine when run on the same machine as a user account. Is there a supported way to configure the uba/horde login so it doesn’t require a browser session? I created a service account but could not find any clear documentation on how to set up the uba config for that account to login.
Hey there Eric,
Our current documentation for UBA authentication is here - specifically the “Horde jobs utilizing remote compilation are automatically granted access via an injected token that’s set as an environment variable for each job step (UE_HORDE_TOKEN)”. You’ll need to make sure you have your acl setup correctly to allow for remote execution.
{
// ...
"plugins": {
// ...
"compute": {
// ...
"clusters": [
{
"id": "default",
"namespaceid": "horde.compute",
"acl": {
"entries": [
{
"claim": { "type": "http://epicgames.com/ue/horde/user", "value": "jane.smith" },
"actions": ["AddComputeTasks"]
}
]
}
}
]
}
}
}
````Could you provide some more details on how auth is currently configured on your end?
Kind regards,
Julian
"Horde jobs utilizing remote compilation are automatically granted access via an injected token that’s set as an environment variable for each job step"
I’m not clear how this automagic injection process works. How does the build machine get authorization initially to have a token auto injected and who injects it? It seems like whatever that token is, it still reverts to the interactive login because I guess it is failing?
Re: auth configuration. I originally tried OIDC with an Azure app. I got that to work right up until the issue I saw posted in another thread where there is a ServerComputeClient failure because the server is processing the token as OIDC but it’s actually an internal Horder server token. I wasn’t clear how to fix this though from that thread. It seems to be related to the (to me) opaque process of this token. Everything else was working on my local machine though, including the UBA horde login step, and normal login to the horde server for interactive/dashboard.
I then switched to “Horde” authentication and I have the ACL definition set up properly, similar to what you posted, and it works perfectly on my local PC. However, as mentioned, it fails on the build machine because it can’t do an interactive login.
Hey Eric,
So I had a conversation with one of the subject matter experts here, and something certainly feels off here. Regarding the injection of UE_HORDE_TOKEN, this occurs from the horde server side… That being said, you should be able to add some debug code to test to see whether the initiating machine (in your instance, the Horde agent’s UBT invocation - from a service account) is getting anything back from that variable.
If you add some debug printouts here (and we can also see that the interactive flag comes into play if we can’t obtain it), and run it (either by pre-submit or by checking in temporarily) you should be able to see whether we are able to get a valid access token. I’d be curious to see whether the service account has a different output than your user account - which I’d presume is the case.
At least from there we could sort out whether the env var is making it’s way to the UBT invocation context.
One current hypothesis is that it’s failing in both cases, but the user account is using a fallback to oidc based auth.
Kind regards,
Julian
> I’m actually not sure how to config the UBT to use the service account. If you can point me to that I will try.
Not UBT specifically to run using the service account, just the horde execution context from earlier where you were running this as a service (presumably under the service account). re: “Okay, so because it is running as a service account…”. The goal here is to see what the UE_HORDE_TOKEN is in both contexts (one that worked - where you had logged in - one that failed - where you were running it as a service (?) presumably).
>I was able to get a different error message by copying the appdata oidcTokenStore.dat from a local interactive login to the system account appdata folder. Instead of failing on the attempt to interactive login it now fails with a…
I haven’t seen this error in the log before. Are there any other items in that log that would highlight what file is missing?
You can also check to make sure you have the UBA binaries @ Engine\Binaries\Win64\UnrealBuildAccelerator\*
Also, just to be clear, what version of Horde are you on?
Hey there Eric,
Just a quick note after conversing a bit with my colleague ([mention removed]) regarding the use case of:
- As a build engineer, I would like to be able to use UBA+Horde from my own Non-horde CI so that I can reduce my build times
As it stands right now, our internal use cases have been iether to use interactive auth, or let things get authed via the horde agent flow (which of course, would place a more stringent requirement on either using Horde, or finding some way to wrap your non-horde context within an horde agent). Now, you can utilize the /api/v1/admin/roletoken and /api/v1/admin/token endpoints to at the very least obtain a Horde server minted token for the !Horde build devices - and set them to the HordeToken UBA config property.
You’ll hit some of the challenges seen [Content removed] in token expiry, with the most poignant feedback being to set the jwtExpiryTimeHours (server config) to some longer interval. This does mean that you’ll need to manage the setup initially, but it should at least unblock in the interim. I’ve also created an internal JIRA (for posterity, internal JIRA is UE-282177) for the user story so that we can better service this (both in documentation, and perhaps an endpoint to better allow users to request a token for some duration). Let me know if obtaining the admin token and stashing it helps in a workaround in the interim.
I just wanted to also say thank you for collaborating on this, and the back and forth. It’s always important for us to hear from our licensees. Receiving feedback on where things are high friction are really important in improving the product.
Kind regards,
Julian
> I had gotten the service processes to work with the approach I documented above -- use interactive login to grab a token which saves into oidctoken.dat. This approach worked multiple times, but it would often need to re-interactive login after only one or two runs instead of successfully refreshing.
- Just reviewing the TryDoRefreshTokenAsync source (OidcToeknManager), the refresh tokens themselves supposedly have a short shelf-life (“refresh tokens are always one time use only so we need to store this new refresh token we got so it can be used the next time”)
- After breaking into the code, I gathered my refresh tokens and ran them through https://jwt.io/
- My expiry is coming back as a week from today - which is a bit surprising given the comment
I’ll run some further tests around the jwtExpiryTimeHours to see how this interacts with the refresh token. If you see any anomalous behaviour on your end, please do add it back to this thread as I’m coalescing all of this into a Knowledge Base article on auth debugging for Horde.
Edit:
- Further tests completed - if a Token is hand presented via the BuildConfiguration.xml <Horde><Token>token from api/v1/admin/token</Token></Horde>, the entire refresh path will be circumvented and TryDoRefreshTokenAsync won’t be invoked, so you should be in the clear on this
Julian
I see this in another thread:
"If you avoid setting any token in BuildConfiguration.xml, the Horde client inside UBT will check with the Horde server what authentication is in use (calls https://<address>/api/v1/server/auth.
With AuthMode=Horde, it will ask you to login as normal to Horde in a browser the first time, then save an access and refresh token to disk. These tokens will automatically renew as needed. In this mode, Horde itself acts as an OpenID Connect provider."
Okay, so because it is running as a service account, presumably it is looking for/saving these in a different location from if I were to run it interactively on the build machine? Which I did and which did succeed.
Also, this feels like an impractical setup to require every build machine to have one interactive login. Is there another approach for this?
I’m actually not sure how to config the UBT to use the service account. If you can point me to that I will try.
I was able to get a different error message by copying the appdata oidcTokenStore.dat from a local interactive login to the system account appdata folder. Instead of failing on the attempt to interactive login it now fails with a
---- Starting trace: 250508_130038 ----
UbaServer - Listening on 0.0.0.0:1345
------ Building 3831 action(s) started ------
Unable to create Horde session: The system cannot find the path specified.
So I stepped through and confirmed there is no UE_HORDE_TOKEN environment variable at that stage. But again, how would it get there? The only method I’ve seen is either to hardcode a token that will expire in the buildgraph or to have it run the auth in that function, which seems to require a login or a pre-existing refresh token that is read from the encrypted .dat file.
5.5.4 is our version.
>> the horde execution context from earlier where you were running this as a service (presumably under the service account). re: “Okay, so because it is running as a service account…”.
okay, so I made a “service account” on the horde server, but this doesn’t sound like what you meant. it is what i was wondering, but sounds not supported or irrelevant. then in my quote about “running as a service account” i mean the windows service and i think you meant that as well, so all good now.
>> I haven’t seen this error in the log before. Are there any other items in that log that would highlight what file is missing?
so this was a misleading message. the actual error was a failure to decrypt the oidctoken.dat file. I managed to fix it by setting CryptProtectFlags flags = CryptProtectFlags.CryptprotectLocalMachine;
in 2 places in WindowsTokenStore.cs. That allowed me to run the interactive login on my user account and copy the token to the service account appdata and all is well.
In no instance do i see the UE_HORDE_TOKEN being set though. I think it always pulls it from the refresh or auth token stored in that encrypted store, and when that doesn’t exist it tries to pop the browser to login.
I still really need to understand what setting or config would allow a token to be injected into the environment prior to this step, because none of the working flows I’ve seen do that.
So “all is well” meaning the service account can now run builds using horde and distributed build. that’s “great” but I don’t think this mechanism of hacking the login and copying the token to the service app data is the way this is supposed to work.
Thanks. So we know that the execute task coming from the server either 1) doesn’t have the token or 2) it’s not making it’s way to the underlying invocation.
> How would it get there.
Potential follow-up is: “who creates it (the token)” - the server, with the code here. This should be trying to use the ACLServiceto mint a new token.
I’ll boot up a quick HordeAuth server from scratch and see if anything jumps out.
Thanks for confirming re:version + service account & that missing file.
I replied above, going to inline here:
> How would it get there.
Potential follow-up is: “who creates it (the token)” - the server, with the code here. This should be trying to use the ACLServiceto mint a new token.
I’ll boot up a quick HordeAuth server from scratch and see if anything jumps out.
Isn’t that on the agent side though? Is that a different way to do this? We are directly running UBT on the build server (in service process) which would be a client, not an agent, right?
I’m seeing this in another thread on getting this to work:
"The token returned via oidctoken.exe (UBA flow) "
That executable project exists but seems to have rotted a bit. That said I still see code in C++ that references it. It seems like the only path I can wrap my head around for this horde token to be pulled from the server to the client. Is it still used? Do I need to set up the oidc-configuration.json file to make this work?
This post has a response from [Carl [Content removed] (Epic Games) that feels like the best description/documentation of the flow in question:
[UBA authentication [Content removed]
Also, to be clear, I’ve used this flow successfully on my local PC.
However, it still puts us right back at the start of this thread, because it only describes the interactive browser login method. So the question still remains: in 5.5 is there a supported method to generate/store a token on a build server that runs CI jobs in a non-interactive (service) mode? If so, what is the secret? 
(and to clarify, a CI job for us is just a command line run of UBT - pretty much the same as we would do on a local PC but with diff params)
Ah, I have a sneaking suspicion on where wires are getting a bit crossed.
I think I’ve made an (incorrect!) assumption here that when you’re referring to CI, you’re implicitly using Horde as the CI mechanism. I suspect I’m incorrect in this - and as a result, you’re not using the HordeAgent (and subsequently, you’re not getting the whole JobTaskSource minting of JIT token, which is then injected through the entire call hierarchy I listed above).
So regarding:
- > Isn’t that on the agent side though? Is that a different way to do this? We are directly running UBT on the build server (in service process) which would be a client, not an agent, right?
Yes this is through the Horde Agent side - and the Horde Build plugin context. Carl has outlined very much the local user context, that is, outside of the HordeAgent (and JobDriver) wrapper, and how that auth path progresses (and where interactive is seeming an “OK” option).
If all tracks up to this point, then I have a more concrete user story that I think I can work with [mention removed] on to see if we can find a path forward for you (some of the referenced EPS thread has attributes that have since been obsoleted - OIDCProvider being one of them).
Let me know if we are aligned.
yep, that’s it. i might investigate horde CI separately, but we’ve got a lot of process around our tools/ci system as it is now, so getting a path for this would be huge.