[Horde] JobDriver aborts every batch/conform when a process has a non-ASCII image path

General Issues and Bug Reporting (Unreal Engine)
1

Summary

Horde / JobDriver (Engine/Source/Programs/Horde), server + agent 5.8.0-54390005, driver 5.8.0.0, Windows 10.0.26200 zh-CN.

On a localized Windows host, if any running process has a non-ASCII executable path (e.g. D:\AI_Project\文言文翻译器\venv312\Scripts\pythonw.exe), JobDriver throws during the process-cleanup step that runs before every batch and every conform, and no step is ever executed:

System.ArgumentOutOfRangeException: Index and length must refer to a location within the string. (Parameter ‘length’)
at System.Text.StringBuilder.ToString(Int32 startIndex, Int32 length)
at EpicGames.Core.ProcessUtils.TerminateProcessesWin32(…) EpicGames.Core\ProcessUtils.cs:line 155
at JobDriver.Execution.JobExecutor.ExecuteBatchAsync(…) JobExecutor.cs:line 531

Cause: the P/Invoke in ProcessUtils.cs:34 has no CharSet, so the ANSI entry point is bound; lpdwSize is then in bytes while the decoded StringBuilder length is in characters:

[DllImport(“kernel32.dll”, SetLastError = true)] // → QueryFullProcessImageNameA
static extern int QueryFullProcessImageName(…, [Out] StringBuilder lpExeName, ref int lpdwSize);

int characters = 260;
StringBuilder buffer = new(characters);
if (QueryFullProcessImageName(handle, 0, buffer, ref characters) > 0)
new FileReference(buffer.ToString(0, characters)); // 54 bytes vs 48 chars → throws

The same pattern appears at ProcessUtils.cs:406-410 (GetProcessImageName).

Result: the agent stays Ok/online and keeps polling, but no job or conform can run. Batches end in ~1s, conforms fail the same way, and the agent gets stuck at pendingConform = true so it is never assigned work.

Suggested fix:
[DllImport(“kernel32.dll”, SetLastError = true, CharSet = CharSet.Unicode)]
… + buffer.ToString(0, Math.Min(characters, buffer.Length))
Also: an inspection failure of a single process should be logged and skipped rather than aborting the batch.

What Type of Bug are you experiencing?

Foundation (C++ Tools, Profiling, & Pipeline)

Steps to Reproduce

  1. Use a Windows host with a non-ASCII ANSI code page (e.g. zh-CN, code page 936).
  2. Start a process whose image path contains non-ASCII characters:
    mkdir C:\tmp\测试
    copy C:\Windows\System32\notepad.exe C:\tmp\测试\notepad.exe
    start “” C:\tmp\测试\notepad.exe
  3. Register a Horde agent on that host (5.8.0-54390005) and put it in a pool that has a workspace (we use identifier TKG-Inc, stream //TaikiGame/main).
  4. Trigger a conform on that agent (Conform button, or PUT /api/v1/agents/{agentId} with requestConform).
  5. Trigger any job that targets that agent.

Expected Result

  • The batch starts: driver logs “Initializing executor…” / “Executing steps…”, and the steps actually run.
  • Conform completes and the agent leaves pendingConform state.
  • If inspecting one unrelated process fails, the driver logs a warning and continues instead of aborting the batch.

Observed Result

  • Driver aborts before any step runs:

    09:10:40 Executing jobId …, batchId b434, leaseId … with executor null
    09:10:40 Error: Exception while executing lease …: Index and length must refer to a location within the string. (Parameter ‘length’)
    09:10:40 Done.
    09:10:40 Driver finished with exit code 0

  • Batch result: state=Complete, error=SyncingFailed, step “Setup Build” state=Skipped, job label outcome=Failure, total duration ~1s.

  • Conform lease fails identically (“execute conform”, exit code 1), so the agent remains pendingConform=true and never gets jobs.

  • Only starts when such a process is running; removing it from the host is the only workaround we found short of patching JobDriver. The batch error “SyncingFailed” is misleading here, as nothing related to Perforce syncing has failed.

Affects Versions

5.8

Platform(s)

Windows

Upload an image

image.png
image.png2180×601 134 KB

For crash reports, include your callstack

System.ArgumentOutOfRangeException: Index and length must refer to a location within the string. (Parameter ‘length’)
at System.Text.StringBuilder.ToString(Int32 startIndex, Int32 length)
at EpicGames.Core.ProcessUtils.TerminateProcessesWin32(Predicate1 predicate, ILogger logger, CancellationToken cancellationToken) in ...\Engine\Source\Programs\Shared\EpicGames.Core\ProcessUtils.cs:line 155 at EpicGames.Core.ProcessUtils.TerminateProcesses(Predicate1 predicate, ILogger logger, CancellationToken cancellationToken) in …\ProcessUtils.cs:line 97
at JobDriver.Utility.TerminateProcessHelper.TerminateProcessesAsync(TerminateCondition condition, DirectoryReference workingDir, IReadOnlyList`1 processesToTerminate, ILogger logger, CancellationToken cancellationToken) in …\Horde\Drivers\JobDriver\Utility\TerminateProcessHelper.cs:line 51
at JobDriver.Execution.JobExecutor.ExecuteBatchAsync(RpcBeginBatchResponse batch, ILogger logger, ILogger localLogger, CancellationToken cancellationToken) in …\JobDriver\Execution\JobExecutor.cs:line 531
at JobDriver.Execution.JobExecutor.ExecuteInternalAsync(ILogger logger, CancellationToken cancellationToken) in …\JobDriver\Execution\JobExecutor.cs:line 494

Additional Notes

  • This is Horde, not an Unreal Editor crash: the failing component is the JobDriver process that the Horde agent launches per lease (C:\Program Files\Epic Games\Horde\Agent\JobDriver\JobDriver.dll, version 5.8.0.0).

  • Trigger condition: the host has at least one running process with a non-ASCII executable path. We confirmed it by comparing the failure timeline with the process list on the agent; the only non-ASCII image path was:
    D:\AI_Project\文言文翻译器\venv312\Scripts\pythonw.exe (48 chars, 54 bytes in code page 936)

  • Consequence: every batch and every conform fails in TerminateProcessesWin32. Batches end in ~1s with error=SyncingFailed and step “Setup Build”=Skipped; conforms fail too, so the agent stays pendingConform=true and is never assigned work, while still reporting Ok/online in the UI.

  • The reported batch error (SyncingFailed) is misleading: nothing related to Perforce syncing failed. It cost us significant time to rule out the workspace/cluster.

  • Workaround: none found other than not having such processes on the agent host, or patching JobDriver (CharSet.Unicode + length clamp in ProcessUtils.cs:34/155 and 406-410).

  • Happy to provide a minimal standalone C# reproduction (P/Invoke QueryFullProcessImageNameA against a non-ASCII path) if that helps.