Horde: What account does UE_HORDE_TOKEN correspond to when a job is running via scheduler?

I made a custom Horde task that uses the /api/v1/agents and /api/v1/jobs endpoints to query a list of agents then run a job on the agent

It uses UE_HORDE_TOKEN to do the request and it runs fine when I manually run the job, and the jobs it spawns say they were run by me

When run via scheduler, it fails with a 403 error, “User does not have ListAgents permission”

I looked at our common.global.json file and in our credentials section, we use account “foo” so I tried updating our acl entries section to go

{ "claim": { "type": "http://epicgames.com/ue/horde/user", "value": "<foo>" }, "profiles": [ "default-run", "default-read" ], "actions": [ "ViewJob", "CreateJob", "CreateAgent", "UpdateAgent", "DeleteAgent", "ViewAgent", "ListAgents", ] },

thinking the scheduler used the default account “foo” to run the jobs but I still get “User does not have ListAgents permission” so I guess I’m granting the permissions to the wrong user

What user should I be granting permissions to?

Steps to Reproduce
* Make job that calls into endpoint

* Run job and see what it works

* Have job run via scheduler and see that it fails

Hey Adam,

Thanks for your question. Can you highlight where it fails? Is this agent side (so the delegated machine) or is this on the server side? Do you have a bit more details on the ListAgents task? I wonder if we are getting a more opaque permissions error when something nested is indeed failing.

I’ll dig in a bit more on my end here to see if I can find out “who” is attempting to auth.

Julian

Hey Adam,

Just a quick note that I am currently attending a conference this week, so I will be prioritizing this as soon as I’m back next week!

Thanks for your patience.

Julian

Hey Adam,

So I have been stepping through the code in the scheduler (server side) to see how the auth is working, and nothing really stands out. You had mentioned that you were adding a custom Horde task - would it be possible to get some details on this (code snippet, etc). I’d like to try and extend on my end to possibly reproduce this locally.

For posterity, my agent is running as a service, and running as the logged in user (default setup via downloaded HordeAgent install via .msi). I am using the Horde Auth as well with the standard acl entries:

"acl": { "entries": [ { "claim": { "type": "http://epicgames.com/ue/horde/group", "value": "View" }, "profiles": [ "default-read" ] }, { "claim": { "type": "http://epicgames.com/ue/horde/group", "value": "Run" }, "profiles": [ "default-run" ] }One thing I’d like to try out is to see if we can model the unit test for this, as this helps greatly with configuration iteration.

Edit:

  • Let me know if I am wrong here, I think I have assumed divergence in “custom horde task”! If so this is what I’m referring to regarding code.
  • UE_HORDE_TOKEN is only really documented for our RemoteCompilation, so I am curious if something is not hooked up correctly.
  • It’s clear we try and use this in the HordeHTTPClient, so I’m curious about how you’re leveraging this to initiate the logic you’re after.

Julian

Hey Adam,

Good stuff! Thanks for this; I’ll give it a whirl and see what comes up - this is a really great use case. The SME for authentication is out until next week - and I’ll likely need to sit down and walk through this with him to see if we are doing anything off base. I’ll dig into it in the meantime and see if I can track anything down.

Again, this is a great use case and I expect more folks to be trying to do similar things in this space so any findings here will be valuable.

Edit:

  • I have seen in the past where there is a difference between running as a service vs task manager; I wonder if you can attempt this job on a specific pool for a machine that is running as service - in order to rule this out.
  • Another thing you can try is from the UAT task, issue the /api/v1/accounts/current (api/v1/admin/token is also interesting - but is admin) GET API call to see whether the UE_HORDE_TOKEN is actually being injected, and the user is considered logged into Horde.
    • For posterity, you can navigate the APIs quite seamlessly to HORDE_URL/swagger/index.html

Julian

Hey there Adam,

Here is a very topical [Content removed] that’s ongoing at the moment with some more details around UE_HORDE_TOKEN. So the token that is stored within there *should* be minted JIT, and issued through the JobDriver to the UAT job. I’ve outlined that process in the referenced thread.

What’s important to note about the token that’s minted (and stored in UE_HORDE_TOKEN) is the following claims it’s attached:

// Create a bearer token for the job executor List<AclClaimConfig> claims = new List<AclClaimConfig>(); claims.Add(HordeClaims.AgentRoleClaim); claims.Add(new AclClaimConfig(HordeClaimTypes.Lease, leaseId.ToString())); claims.Add(new AclClaimConfig(HordeClaimTypes.LeaseStream, streamConfig.Id.ToString())); claims.Add(new AclClaimConfig(HordeClaimTypes.LeaseProject, streamConfig.ProjectConfig.Id.ToString())); claims.Add(new AclClaimConfig(HordeClaimTypes.LeaseTemplate, job.TemplateId.ToString())); claims.AddRange(job.Claims.ConvertAll(x => new AclClaimConfig(x.Type, x.Value)));Source: JobTaskSource

A couple of ideas here in how to tune the ACL as ended up having to do something similar for a different UGS workflow issue:

  • Attach a debugger to the server (AgentsController) route: “api/v1/agents”
  • Run your scheduled job
    • Catch - in debugger, and inspect the User, and importantly the claims
      • This is where I suspect the above code reference to be relevant here; I wonder if the token that’s being used doesn’t have applicable claims for the specific task to be explicitly called.

Now in researching this other issue, I came across a bug fix that the great [mention removed]​ recently made that *could* be related here - although this is more along the lines of local user context initiating the UBT build in an auth context. It still doesn’t explain the fact that your CI invoked job is failing to pass auth, as for me it was the inverse issue that I observed. That being said we will probably get some insights from the debugger.

I haven’t had an opportunity yet to try and reproduce the above scripts just yet, but my debugging approach would be as above (and mileage may vary). Is this in AuthMethod as Horde? Furthermore, I may have missed a relevant detail from above: when you’re referring to globals credentials - this segment? If so, this will be perforce user which wouldn’t be what you’re running as locally.

Let me know how things fare - and I’m happy to keep chipping away till we get this. I’m planning on coalescing all these notes together for a public article, as Auth can be a bit challenging to get working in some of these flows.

Kind regards,

Julian

Yeah alternatively you can perhaps add some logging to the controller and just dump the claims to the log. It could get a *bit* verbose, but probably easier than attaching via debugger.

If you can add it in the BuildAclModifiers, that’d be a pretty quick and fast way to test if the ACL claims are just missing that in the default (and as a result, for the agent).

Hey there Adam,

Thanks for circling back on this, and it’s certainly my pleasure to help. I’ll suggest this as a user story to the team, as I can see folks wanting to have a bit more control over this without divergence.

Kind regards,

Julian

As an extra data point, one of our jobs is able to call into the /api/v2/devices endpoint when run via scheduler but /api/v1/agents fails

> Thanks for your question. Can you highlight where it fails? Is this agent side (so the delegated machine) or is this on the server side? Do you have a bit more details on the ListAgents task? I wonder if we are getting a more opaque permissions error when something nested is indeed failing.

This fails on the agent side, when calling into the agents endpoint the server responds with a 403 “User does not have ListAgents permission” error when the job gets run via scheduler

> I’ll dig in a bit more on my end here to see if I can find out “who” is attempting to auth.

Thanks! I’m wondering if there’s some default service account that’s missing the listagents permission or something since it works when manually run and is able to call into the devices endpoint when run via scheduler

Thanks for the update, enjoy the conference!

Hi Julian,

We have a buildgraph that calls into RunJobsOnAllAgents.cs. Our agent isn’t running as a service but rather running the exe via task scheduler but don’t see how that could affect things

`<?xml version='1.0' ?>

`

`using UnrealBuildBase;
using Newtonsoft.Json;
using System.Net.Http;
using System;
using System.Collections.Generic;
using Gauntlet;
using System.Text;

namespace AutomationTool.Tasks
{
///


/// Run given job on all agents in given pool
///

[Help(“Run given job on all agents in given pool”)]
[ParamHelp(“targetJobId=”, “Job id to run”)]
[ParamHelp(“agentPool=”, “Pool to run job against”)]
[ParamHelp(“hordeServerUrl=”, “Horde Server URL”)]

public class RunJobOnAllAgents : BuildCommand
{
///


/// Entrance point for command
///

///
public override ExitCode Execute()
{
string targetJobId = ParseParamValue(“targetJobId”, String.Empty);
string agentPool = ParseParamValue(“agentPool”, String.Empty);
string hordeServerUrl = ParseParamValue(“hordeServerUrl”, String.Empty);

Dictionary<string, string> requiredArgs = new Dictionary<string, string>
{
{ nameof(targetJobId), targetJobId },
{ nameof(agentPool), agentPool },
{ nameof(hordeServerUrl), hordeServerUrl },
};

if (hordeServerUrl.EndsWith(“/”, StringComparison.OrdinalIgnoreCase))
{
hordeServerUrl = hordeServerUrl.TrimEnd(‘/’);
}

foreach (KeyValuePair<string, string> kvp in requiredArgs)
{
if (String.IsNullOrEmpty(kvp.Value))
{
Log.Error($“Error: Missing required arg ‘{kvp.Key}’”);
return ExitCode.Error_Arguments;
}
}

using (HttpClient client = new HttpClient())
{
client.Timeout = TimeSpan.FromMinutes(30);
client.DefaultRequestHeaders.ConnectionClose = true;
string token = Environment.GetEnvironmentVariable(“UE_HORDE_TOKEN”);

if (String.IsNullOrEmpty(token))
{
Log.Error(“Error: Couldn’t find Horde token”);
return ExitCode.Error_Unknown;
}
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue(“Bearer”, token);

try
{
string agentsUrl = $“{hordeServerUrl}/api/v1/agents?poolId={agentPool}&includeDeleted=false&invalidateCache=false”;
HttpResponseMessage agentsResponse = client.GetAsync(agentsUrl).GetAwaiter().GetResult();

if (!agentsResponse.IsSuccessStatusCode)
{
string errorContent = agentsResponse.Content.ReadAsStringAsync().GetAwaiter().GetResult();
Log.Error($“Error fetching agents: {agentsResponse.StatusCode} - {agentsResponse.ReasonPhrase}”);
Log.Error($“Error details: {errorContent}”);
return ExitCode.Error_Unknown;
}

string agentsJson = agentsResponse.Content.ReadAsStringAsync().GetAwaiter().GetResult();

List<Dictionary<string, object>> agentsList = JsonConvert.DeserializeObject<List<Dictionary<string, object>>>(agentsJson);
Log.Info($“Found {agentsList.Count} agents in pool {agentPool}”);

string jobsUrl = $“{hordeServerUrl}/api/v1/jobs”;
string streamId = Environment.GetEnvironmentVariable(“UE_HORDE_STREAMID”);
string change = Environment.GetEnvironmentVariable(“uebp_CL”);

foreach (Dictionary<string, object> agent in agentsList)
{
string agentName = agent[“name”].ToString();
bool enabled = Convert.ToBoolean(agent[“enabled”]);
bool statusOk = agent[“status”].ToString() == “Ok”;
bool online = Convert.ToBoolean(agent[“online”]);

Log.Info($“agentName: {agentName}, enabled: {enabled}, statusOk: {statusOk}, online: {online}”);

if (!enabled || !statusOk || !online)
{
continue;
}

var jobPayload = new
{
streamId = streamId,
templateId = targetJobId,
priority = “Normal”,
parameters = new { },
updateIssues = false,
selectedAgent = agentName,
change = Int32.Parse(change)
};

string jobJson = JsonConvert.SerializeObject(jobPayload);
StringContent content = new StringContent(jobJson, Encoding.UTF8, “application/json”);

Log.Info($“Sending POST Url: {jobsUrl}, payload: {jobJson}”);
HttpResponseMessage jobResponse = client.PostAsync(jobsUrl, content).GetAwaiter().GetResult();

if (!jobResponse.IsSuccessStatusCode)
{
string errorContent = jobResponse.Content.ReadAsStringAsync().GetAwaiter().GetResult();
Log.Error($“Error posting job for agent {agentName}: {jobResponse.StatusCode} - {jobResponse.ReasonPhrase}”);
Log.Error($“Error details: {errorContent}”);
Log.Error($“Request payload: {jobJson}”);
}
else
{
Log.Info($“Posted job for agent {agentName}”);
}
content.Dispose();
}

return ExitCode.Success;
}
catch (Exception e)
{
Log.Error($“Unexpected Error: {e.Message}”);
return ExitCode.Error_Unknown;
}
finally
{
client.Dispose();
}
}
}
}
}`

Dang, saw your message before the edit so missed the additional questions

Going

`using (HttpClient client = new HttpClient())
{
client.Timeout = TimeSpan.FromMinutes(30);
client.DefaultRequestHeaders.ConnectionClose = true;
string token = Environment.GetEnvironmentVariable(“UE_HORDE_TOKEN”);

if (String.IsNullOrEmpty(token))
{
Log.Error(“Error: Couldn’t find Horde token”);
return ExitCode.Error_Unknown;
}
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue(“Bearer”, token);

try
{
string accountUrl = $“{hordeServerUrl}/api/v1/accounts/current”;
HttpResponseMessage accountResponse = client.GetAsync(accountUrl).GetAwaiter().GetResult();
string responseContent = accountResponse.Content.ReadAsStringAsync().GetAwaiter().GetResult();
Log.Info($“Account information: {responseContent}”);
…`gets me

> Account information: {“time”:“2025-05-09T17:40:24”,“level”:“Error”,“message”:“User is not logged in through a Horde account”,“format”:“User is not logged in through a Horde account”}

Is the expectation that I’m logged in when using Horde token for auth?

No luck running as a service

hmm I’ll try to breakpoint and step through code on Monday, debugging Horde server is kind of tricky for us, seems plausible that it’s missing claim there.

I see BuildACLModifiers goes

acl.AddCustomRole(HordeClaims.AgentRoleClaim, new[] { ProjectAclAction.ViewProject, StreamAclAction.ViewStream, LogAclAction.CreateEvent, AgentSoftwareAclAction.DownloadSoftware });

I wonder if I need to tack on CreateJob/ListAgents to there

Can confirm that modifying BuildAclModifier.cs to have ListAgents and CreateJob claims did the trick

acl.AddCustomRole(HordeClaims.AgentRoleClaim, new[] { ProjectAclAction.ViewProject, StreamAclAction.ViewStream, LogAclAction.CreateEvent, AgentSoftwareAclAction.DownloadSoftware, AgentAclAction.ListAgents, JobAclAction.CreateJob }

thanks for all the help Julian! now I have a bit of a better understanding of Horde auth stuff