Horde: api/v1/schema/ endpoints crash due to OTel reflection error

Development Pipeline & Plugins
, ,
1

SchemaController endpoints appear to fail due to issues with reflecting on OTel assemblies

{
  "time": "2026-06-24T16:52:23",
  "level": "Error",
  "message": "Unhandled exception: The type initializer for 'HordeServer.Server.SchemaController' threw an exception.",
  "format": "Unhandled exception: {Message}",
  "properties": {
    "Message": "The type initializer for 'HordeServer.Server.SchemaController' threw an exception."
  },
  "exception": {
    "message": "The type initializer for 'HordeServer.Server.SchemaController' threw an exception.",
    "trace": "   at HordeServer.Server.SchemaController.get_ConfigSchemas() in /home/horde/work/ue5_devinfra/Sync/Engine/Source/Programs/Horde/HordeServer/Server/SchemaController.cs:line 40\n   at HordeServer.Server.SchemaController.GetCatalog() in /home/horde/work/ue5_devinfra/Sync/Engine/Source/Programs/Horde/HordeServer/Server/SchemaController.cs:line 77\n   at lambda_method5316(Closure, Object, Object[])\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeActionMethodAsync\u003Eg__Logged|12_1(ControllerActionInvoker invoker)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeNextActionFilterAsync\u003Eg__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()\n--- End of stack trace from previous location ---\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeNextResourceFilter\u003Eg__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()\n--- End of stack trace from previous location ---\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeAsync\u003Eg__Logged|17_1(ResourceInvoker invoker)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeAsync\u003Eg__Logged|17_1(ResourceInvoker invoker)\n   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)\n   at HordeServer.Server.ExceptionLoggingMiddleware.InvokeAsync(HttpContext context) in /home/horde/work/ue5_devinfra/Sync/Engine/Source/Programs/Horde/HordeServer/Server/ExceptionController.cs:line 36\n   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.\u003CInvoke\u003Eg__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)",
    "innerException": {
      "message": "Unable to load one or more of the requested types.\nDerived method 'TryCreateLogger' in type 'OpenTelemetry.Logs.LoggerProviderSdk' from assembly 'OpenTelemetry, Version=1.0.0.0, Culture=neutral, PublicKeyToken=7bd6737fe5b67e3c' cannot reduce access.",
      "trace": "   at System.Reflection.RuntimeModule.GetTypes(QCallModule module, ObjectHandleOnStack retTypes)\n   at System.Reflection.RuntimeModule.GetTypes(QCallModule module, ObjectHandleOnStack retTypes)\n   at System.Reflection.Assembly.GetTypes()\n   at HordeServer.Server.SchemaController.FindSchemaTypes() in /home/horde/work/ue5_devinfra/Sync/Engine/Source/Programs/Horde/HordeServer/Server/SchemaController.cs:line 54\n   at HordeServer.Server.SchemaController..cctor() in /home/horde/work/ue5_devinfra/Sync/Engine/Source/Programs/Horde/HordeServer/Server/SchemaController.cs:line 40",
      "innerExceptions": []
    },
    "innerExceptions": []
  }
}

The symptoms match a known OTel library compatibility issue from 2024: https://github.com/open\-telemetry/opentelemetry\-dotnet/issues/5741

2

Steps to Reproduce

  1. GET the endpoint /api/v1/schema/catalog.json

500 error is returned instead of schema list

3

Hello, I’ve been unable to reproduce this with the latest horde-server container image (5.8).

Could you please confirm which version you’re using, and if 5.8, whether you’re running 5.8.0-preview-1 or 5.8.0-release?

I did see that we encountered runtime issues with OpenTelemetry, and in 5.8.0-release, we bumped the OpenTelemetry.Api version from 1.8.1 to 1.15.3.

https://github.com/EpicGames/UnrealEngine/commit/74d4916d28ddece67b190f14745057016ed08c44

4

We’re on 5.8.0 release, which uses 1.8.1.

In `//UE5/Main` it looks like all the 3rd party dependencies of HordeServer are now not version restricted. That’s probably also not a good idea.

I don’t mind version bumping for just our local deployment in the meantime.

5

Apologies for the misinformation; the commit that I linked did not bump the HordeServer OpenTelemetry version, and it is still 1.8.1 on 5.8.0-release, as you mentioned.

https://github.com/EpicGames/UnrealEngine/blob/5\.8\.0\-release/Engine/Source/Programs/Horde/HordeServer/HordeServer.csproj\#L45

Regarding the versioning change, we are moving to centralized NuGet package versioning for all non-third-party .NET projects.

This is why the versions will no longer sit with the Horde projects; instead, they can be found in <root-directory>/Directory.Packages.props.

6

Ah, interesting. Moving to Directory.Packages.props is definitely a much easier system.

Thanks!