Intermittent UBA Horde failures during shader compilation in cook

Hi,

We’re currently integrating UBA for both C++ and shader compilation in Unreal Engine 5.6, and we’re hitting intermittent failures during shader compilation in cook.

Context

  • Unreal Engine version: 5.6
  • Horde / UBA agents: 5.7
  • UBA enabled for:
    • C++ compilation (working reliably)
    • Shader compilation (enabled, but unstable)
  • Cook duration: ~2 hours
  • Issue is intermittent, the same build often succeeds on retry

Configuration

BuildConfiguration.xml

<bAllowUBA>true</bAllowUBA>
<bAllowUBAShaderCompile>true</bAllowUBAShaderCompile>

DefaultEngine.ini

[Uba.Provider.Horde]
Enabled=true
ServerUrl="<https://can-horde-server>..."
Pool=BuildMachine
Cluster=default
MaxCores=2048
 
[UbaController]
Providers=Uba.Provider.Horde
bReuseProcesses=True
 
[DevOptions.Shaders]
bAllowCompilingThroughWorkers=True
bAllowAsynchronousShaderCompiling=True
bAllowDistributedShaderCompiling=True
bUseXGEController=False
bUseHordeShaderCompiler=True
 
MaxShaderJobBatchSize=16
WorkerTimeToLive=600
NumUnusedShaderCompilingThreads=2

Observed Behavior

At the beginning of the cook, UBA + Horde appears to initialize correctly:

LogUbaHorde: UBA/Horde Configuration:
  Server: <https://can-horde-server>...
  Pool: BuildMachine

We also see:

LogShaderCompilers: Display: Using UBA Controller for shader compilation
LogUbaController: Starting up UBA/Horde connection for session ...

However, later during the cook:

LogUbaController: Shutting down UBA/Horde connectionThen shortly after:

LogUbaController: Starting up UBA/Horde connection for session ...
LogUbaHorde: Warning: Getting Horde server URL failed [Source: Not found]
LogUbaHorde: Error: Failed to create HttpClient for UbaAgent

At this point the cook fails.

What we observed

  • The same build often succeeds on retry
  • The failure happens after a successful initial Horde session
  • The error occurs during re-initialization of UBA/Horde
  • C++ compilation via UBA/Horde is stable
  • Issue appears only in shader compilation during cook
  • Logs suggest the ServerUrl becomes unavailable on second initialization

What we tried

  • Fixed INI parsing issues (quotes around ServerUrl)
  • Explicit

[UbaController] Providers=...* Enabled

bUseHordeShaderCompiler=True* Increased

WorkerTimeToLive* Enabled

bReuseProcesses=True* Cleared Saved/Intermediate configs

  • Verified Horde pool and agents are valid

Suspicions

We suspect one (or a combination) of the following:

  1. UbaController lifecycle issue in UE 5.6
  • Config not reloaded on second initialization
  1. Version mismatch
  • UE 5.6 client vs Horde/UBA agents 5.7
  1. Shader compilation path being less stable than UBT path
  2. Session teardown/recreation during long cooks causing loss of Horde config

Questions

  1. Is UE 5.6 expected to be stable for UBA + Horde shader compilation during cook?
  2. Are client/agent version mismatches (5.6 vs 5.7) known to cause this kind of issue?
  3. Should the Horde config be reloaded on every UbaController initialization?
  4. Is there a known fix or patch for this in 5.6?
  5. Are there recommended settings to prevent UbaController teardown during long cooks?

Any guidance would be greatly appreciated, especially if this is a known issue or something already fixed in later versions. Happy to provide full logs or test specific changes if needed.

Thanks!

Nathaël Goudreau

PS : Appologies for the duplicate post, I posted in https://forums.unrealengine.com/ before this post.

[Attachment Removed]

Steps to Reproduce
Launch cook with UBA enabled for shader compilation

Observe that UBA Controller was properly initialized in the log

LogUbaHorde: UBA/Horde Configuration:
  Server: <https://can-horde-server>...
  Pool: BuildMachine

LogShaderCompilers: Display: Using UBA Controller for shader compilation
LogUbaController: Starting up UBA/Horde connection for session ...

Observe later in the cook process that UBA Controller shuts down

LogUbaController: Shutting down UBA/Horde connection

Observe that UBA Controller fails to reinitialize with error : Failed to create HttpClient for UbaAgent and fails the cooking

LogUbaController: Starting up UBA/Horde connection for session ...
LogUbaHorde: Warning: Getting Horde server URL failed [Source: Not found]
LogUbaHorde: Error: Failed to create HttpClient for UbaAgent

[Attachment Removed]

Hey there,

There is a lot to dig into here, but one thing that wasn’t clear is do you have a [Horde] section with serverurls?

There could very-well be some mismatch between 5.6 and 5.7 here.

Julian

[Attachment Removed]