Reservation Error in Horde

Development Pipeline & Plugins
, ,
1

Hello, I am continuing to test Horde for integration into our pipelines. Today I encountered an issue while reserving a device through the Device Manager.

My request in swagger:

/api/v2/devices/reservations { "poolId": "ci", "devices": [ { "platformId": "playstation" } ]In response, I receive the following error:

source

{ "time": "2025-06-09T16:26:27", "level": "Error", "message": "Unhandled exception: Object reference not set to an instance of an object.", "format": "Unhandled exception: {Message}", "properties": { "Message": "Object reference not set to an instance of an object." }, "exception": { "message": "Object reference not set to an instance of an object.", "trace": " at HordeServer.Devices.DevicesController.CreateDeviceReservationAsync(CreateDeviceReservationRequest request) in /app/Source/Programs/Horde/Plugins/Build/HordeServer.Build/Devices/DevicesController.cs:line 549\n at lambda_method9063(Closure, Object)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask1 actionResultValueTask)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)\n at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)",
“innerExceptions”:
}
}`Despite the error, the reservation is made; however, I do not receive a proper response, which means I don’t have access to the reservation ID (and I assume the IP address should also be returned).

We are using the latest version of the assembled container.

I can build my own container with added logging, but perhaps this is a known issue?

And one more question: is there a way to restrict the permission to create new entries in the Device Manager? Even with zero claims, I was still able to add new devices.

Best regards,

Edward

2

Hey there,

Thank you for this report! Since it seems that you’re in a local testing environment, you can actually attach a debugger to your server instance to see what’s coming up null here. Let me know if that’s at all possible. Can you also share your device manager configuration? FWIW, there’s nothing too extravagant going on with Horde’s device manager - as it’s effectively a keyvalue pair to list kits, and track if they’re in use or not.

>And one more question: is there a way to restrict the permission to create new entries in the Device Manager? Even with zero claims, I was still able to add new devices.

You should be able to control this via the DeviceWrite ACL Action (Action list here). There isn’t specific pool controls per-se, but more so project scoped pools (and the resulting ACL for that project), and for global, you require either admin or “Internal-Employees” user claim role. In short:

  • You must have DeviceRead or DeviceWrite ACLs on at least one project in the pool’s ProjectIds (Project Scope)
  • You must have “Internal-Employees” user claim role or have AdminWrite.

To review a user’s claim/entitlement set, you can do so in the User profile (or via /api/v1/account/entitlements). This is of course assuming you have authentication enabled.

Kind regards,

Julian

3

Hi.

Unfortunately, I’m not deploying the environment locally, but rather in a Docker container on Google Cloud. Connecting a debugger is not that straightforward, and I was hoping this was a known issue so I wouldn’t have to do it myself. I’ll try to figure out how to set it up.

As for the device manager configuration, I’m not entirely sure which settings you’re interested in. I just added a few platforms and two pools of different types for testing.

"plugins": { "build": { "projects": [], "perforceClusters": [], "devices": { "platforms": [ { "id": "playstation", "name": "PlayStation", "models": [ "PS5", "PS5 Pro", "PS4 Slim", "PS4 Pro" ] }, { "id": "xbox", "name": "Xbox", "models": [ "Xbox One", "Xbox Series X", "Xbox Series S" ] } ], "pools": [ { "id": "ue5", "name": "UE5", "poolType": "Shared" }, { "id": "ci", "name": "CI/CD", "poolType": "Automation" }, ] } }, },

4

I couldn’t connect to the remote Docker quickly, so I ran it locally and reproduced the issue. ModelId == null.

I just started the horde server in debug configuration, added one device to the UE5 pool with platform id == android. Then I made a reservation request from the UE5 pool.

[Image Removed]

5

Hey there,

No I have not seen this reported either internally, or externally (although I’m not deeply engaged with or familiar with the specific devices API). If you copy and paste your configurations over to a local setup you should be able to verify quite quickly against a local installation (or even visual studio launched).

I’ll try locally to see what’s up as well with my debug setup.

Edit: Just saw your new post - thanks for giving that a whirl. Will try your config locally - stay posted. The modelID is null coalesced, so I doubt it’s that, but I’ll look.

Julian

6

Just looking at the code a bit more, it looks like it’s the Devices list being null, and it’s unclear to me where this is actually initialized… Let me circle over to the SME for this code.

Julian

7

I really didn’t look closely enough. I double-checked and you’re right: the list simply wasn’t initialized. I think this fix is sufficient. Or you could just add a constructor for CreateDeviceReservationResponse.

[Image Removed]

8

Yup - the one thing I wanted to double check with the SME is why we haven’t seen wide issues with such an obvious NPE. Looks like internally we only use the V1 api, so this would explain it.

For posterity, we’ve deployed a fix internally (once it replicates on Git I can circle back). Much appreciated for bringing this to our attention.

Kind regards,

Julian

9

Indeed they do. I recently posted a practical debugging guide that highlights auth a bit more. It’s a pretty complicated topic, and there’s references from there regarding specific paths for specific tokens (there are a lot of different tokens involved, even just from the Agent perspective). It also covers how you can get a server minted auth token with longer TTL for experimentation.

Kind regards,

Julian

10

I’d also like to add that I ran into difficulties when building the Horde image on Windows using Podman.

I ran the build graph:

RunUAT.bat BuildGraph -Script=Engine/Source/Programs/Horde/BuildHorde.xml -Target="Build Bundled Docker Image"

I got the error:

Running: C:\Program Files\RedHat\Podman\podman.exe cp horde-installer-container:/app/Dashboard/dist/. "I:\Programs\UnrealEngine\Source\UE_5/Staging/Dashboard" Error: container "I" does not exist Took 0,53s to run podman.exe, ExitCode=125 podman terminated with an exit code indicating an error (125) while executing task <Docker Arguments="cp horde-installer-container:/app/Dashboard/dist/. &quot;I:\Programs\UnrealEngine\Source\UE_5/Staging/Dashboard&quot;" />The cause was that Podman interprets the “:” character as a separator (<container>:<path inside сontainer>)

To fix this, I had to modify the build graph by replacing:

<Docker Arguments="cp horde-installer-container:/app/Dashboard/dist/. &quot;$(StagingDir)/Dashboard&quot;"/>with:

<Docker Arguments="cp horde-installer-container:/app/Dashboard/dist/. Dashboard" WorkingDir="$(StagingDir)"/>It’s a bit tricky, but it works. I think Docker might handle arguments differently, so other developers may not have run into this. Unfortunately, I currently don’t have the means to verify this with Docker.

Best regards,

Edward

11

I have another question, and instead of starting a new thread, I’ll ask it here:

What is the intended pipeline for build agents to authenticate in Horde? I’m particularly interested in the device reservation functionality. We’ve set up Google authentication for users, but it’s not entirely clear to me how to handle automation. We can issue a bearer token for an authenticated user and use it afterwards, but these tokens always have a limited lifetime. Is there a way to use OpenID authentication for automation?

Best regards,

Edward

UPD.

I see…

Note that you need to specify "ServiceAccount XYZ" in the Authorization header rather than "Bearer XYZ"

Turns out service accounts do work :slight_smile:

I was beginning to think that with OIDC authentication, other authentication methods wouldn’t work.