Since updating to UE5.6, I can no longer use UBA to compile my projects. Everything was (and still is) working fine with projects under UE5.5.
When compiling my project with 5.6, I encounter numerous errors during remote execution. For example:
Unhandled Exception: System.IO.FileNotFoundException: The specified module could not be found. (Exception from HRESULT: 0x8007007E) ASSERT: Unhandled exception/crash. Suppress debugger startup and try to report issue instead. This message is here to hopefully see callstack CALLSTACK: uba::Detoured_CreateProcessW (UbaDetoursFunctionsKernelBase.inl:3050) KERNEL32.dll: +0x71b3c KERNEL32.dll: +0x72631 KERNEL32.dll: +0x72136 KERNELBASE.dll: +0x182f4b ntdll.dll: +0xaa4bd ntdll.dll: +0x8f827 ntdll.dll: +0xa5e1f ntdll.dll: +0x1e8a6 ntdll.dll: +0x54975 KERNELBASE.dll: +0x601dc clr.dll: +0x181058 11>EXEC: Error: Process 2664 VCTIP.EXE (D:\HordeAgent\Sandbox\Saved\Uba\sessions\250617_120642\bin\fa304d3f3fd8401ec2b3f57b74bb562d\VCTIP.EXE) not active but did not get exit message. Received 18 messages (GetExitCodeProcess returned 0xe0434352)or
11>c1xx: Error C1356: unable to find mspdbcore.dll
There are additional errors, and I also see many log lines like these (I’m not sure if they are related):
WSAIoctl SIO_TCP_INFO failed (The attempted operation is not supported for the type of object referenced.)or
Get the engine source from P4 (stream //UE5/Release-Latest)
Compile Unreal Horde with BuildGraph:
.\Engine\Build\BatchFiles\RunUAT.bat BuildGraph -script="Engine\Source\Programs\Horde\BuildHorde.xml" -target="Build Horde Server Installer" -set:SignOutputs=false* Update the registry to add the new build engine:
`Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Software\Epic Games\Unreal Engine\Builds]
“5.6_EpicPerforce”=“D:\EpicGames\UE5_Release-Latest”`* Install UnrealHordeServer.msion a Windows Server.
Connect to the Horde Server dashboard to download and install the Unreal Toolbox.
Install the Horde Agent from the Toolbox.
Update BuildConfiguration.xml to enable UBA and set the correct Horde Server host.
Create a new project using any C++ template.
Associate the .uproject file with the correct engine version.
Open and recompile the solution in your IDE (Rider, in this case).
I haven’t uploaded the repro project since I only used a C++ template project, but I’ve enclosed the BuildConfiguration.xml file and the log files generated by UBT.
Thanks for reporting this. A quick test that you could run is recompiling/obtaining the binaries from UE5 main (//UE5/Main/Engine/Binaries/Win64/UnrealBuildAccelerator/x64/) as we did have some issues for 5.6 (and are preparing for 5.6.1). If this also fails in your setup, would it be possible for you to try this again in debug, and provide the logs. The process is outlined here.
I’ve tried using UBA build from UE5 Main stream (//UE5/Main), and while it’s a bit better, it’s still failing.
The strange log lines are gone, but I’m now encountering “Access violation reading at address” errors as well as some file access issues.
I’ve attached the logs (with UBA in Debug mode), using the exact same setup and project (UE5.6 from //UE5/Release-Latest + ThirdPerson c++ template project).
Can you confirm your Windows version? There has been a related issue that looks quite similar here [Content removed] [Content removed] If this matches up, you may need to get the referenced binaries (5.5; 5.6 are available from a recompile on the UE5-main).
You’re correct in that you shouldn’t need to recompile if binaries from UE5-main. You will however need to update the TOML file (or create it) to disable the custom malloc path which could cause this.
I’m still occasionally encountering the following errors:
Unhandled Exception: System.IO.FileNotFoundException: The specified module could not be found. (Exception from HRESULT: 0x8007007E) ASSERT: Unhandled exception/crash. Suppress debugger startup and try to report issue instead. This message is here to hopefully see callstack CALLSTACK: uba::Detoured_CreateProcessW (UbaDetoursFunctionsKernelBase.inl:3063) KERNEL32.dll: +0x782d2 KERNEL32.dll: +0x511b2 KERNEL32.dll: +0x7893d KERNELBASE.dll: +0x11a4ec ntdll.dll: +0x16ab7b ntdll.dll: +0x1271a3 ntdll.dll: +0x1666af ntdll.dll: +0x89cd7 ntdll.dll: +0x88b66 KERNELBASE.dll: +0xc9f0a clr.dll: +0x181058However, they’re not causing the build to fail, and everything completes successfully in the end, so I suppose it’s fine?
We are seeing very similar issues here starting with 5.6. Most commonly, hundreds of intermittent errors like this:
UbaServer - WSAIoctl SIO_TCP_INFO failed (The attempted operation is not supported for the type of object referenced.)These errors don’t actually fail the compilation, but being errors, they fail the enclosing Horde Job. They don’t occur deterministically, but it’s a high percentage of our builds.
hmmm, vctip.exe should not be spawned from uba. I actually submitted some fixes today/yesterday that fixes that. would it be possible for you to test building your own binaries from latest? ([mention removed]​ our farm has built new binaries since an hour or two back, I don’t know if they end up on github?)
They would, but it likely resides in bulk data (so they would need to run the Setup.bat) from the ue5-main branch, which would obtain this bulk data. To make everyone’s life a bit easier, I’ve attached the latest from UE5-Main UBA Binaries, X64 @ 44175145.
Hello everyone, @Honk and @JulianGamble. I’ve also been experiencing very similar compile issues with regards to trying to build unreal engine from source with my host machine, uba, horde, and aws ec2. Like OP suggested, the build does finish successfully, although I have to build again a second time to fix the modules that couldn’t be compiled on the horde agents. Note that I’m building the release branch which at the time is 5.6.1, not the ue5-main branch. And I edit the UbaHost.toml file like someone on this thread suggested but it didn’t seem to make a difference (unless I’m supposed to rerun Setup.bat or GenerateProjectFiles.bat after?) Here are some of the errors I saw in Visual Studio on my host machine:
12>[11/7931] Compile [x64] MiMalloc.c [RemoteExecutor: Worker1]
12>
12>Unhandled Exception: System.IO.FileNotFoundException: The specified module could not be found. (Exception from HRESULT: 0x8007007E)
12>ASSERT: Unhandled exception/crash. Suppress debugger startup and try to report issue instead. This message is here to hopefully see callstack
12> CALLSTACK:
12> uba::Detoured_CreateProcessW (UbaDetoursFunctionsKernelBase.inl:3052)
12> KERNEL32.dll: +0x78322
12> KERNEL32.dll: +0x511b2
12> KERNEL32.dll: +0x7898d
12> KERNELBASE.dll: +0x119e4c
12> ntdll.dll: +0x16a01f
12> ntdll.dll: +0x126553
12> ntdll.dll: +0x166fef
12> ntdll.dll: +0x74557
12> ntdll.dll: +0x733e6
12> KERNELBASE.dll: +0xc7f7a
12> clr.dll: +0x1810a8
12>EXEC : error : Process 2616 VCTIP.EXE (C:\HordeAgent\Sandbox\Saved\Uba\sessions\250910_052612\bin\176150b73b0c7791902a1aaa066cb3ac\VCTIP.EXE) not active but did not get exit message. Received 23 messages (GetExitCodeProcess returned 0xe0434352)
12>[88/7931] Compile [x64] Module.Core.8.cpp [RemoteExecutor: Worker0]
12>
12>Unhandled Exception: System.IO.FileNotFoundException: The specified module could not be found. (Exception from HRESULT: 0x8007007E)
12>ASSERT: Unhandled exception/crash. Suppress debugger startup and try to report issue instead. This message is here to hopefully see callstack
12> CALLSTACK:
12> uba::Detoured_CreateProcessW (UbaDetoursFunctionsKernelBase.inl:3052)
12> KERNEL32.dll: +0x78322
12> KERNEL32.dll: +0x511b2
12> KERNEL32.dll: +0x7898d
12> KERNELBASE.dll: +0x119e4c
12> ntdll.dll: +0x16a01f
12> ntdll.dll: +0x126553
12> ntdll.dll: +0x166fef
12> ntdll.dll: +0x74557
12> ntdll.dll: +0x733e6
12> KERNELBASE.dll: +0xc7f7a
12> clr.dll: +0x1810a8
12>EXEC : error : Process 3708 VCTIP.EXE (C:\HordeAgent\Sandbox\Saved\Uba\sessions\250910_052612\bin\176150b73b0c7791902a1aaa066cb3ac\VCTIP.EXE) not active but did not get exit message. Received 18 messages (GetExitCodeProcess returned 0xe0434352)
12>Compile [x64] Module.MeshModelingToolsEditorOnly.cpp [RemoteExecutor: Worker0]: Exited with error code 2 (Force local retry). This action will retry locally
12>Module.MeshModelingToolsEditorOnly.cpp
12>c1xx : fatal error C1356: unable to find mspdbcore.dll
12>Compile [x64] Module.TedsAlerts.cpp [RemoteExecutor: Worker0]: Exited with error code 2 (Force local retry). This action will retry locally
12>Module.TedsAlerts.cpp
12>c1xx : fatal error C1356: unable to find mspdbcore.dll
12>Compile [x64] Module.VPSettings.cpp [RemoteExecutor: Worker0]: Exited with error code 2 (Force local retry). This action will retry locally
12>Module.VPSettings.cpp
12>c1xx : fatal error C1356: unable to find mspdbcore.dll
12>Compile [x64] Module.TedsAlerts.cpp [RemoteExecutor: Worker0]: Exited with error code 2 (Force local retry). This action will retry locally
12>Module.TedsAlerts.cpp
12>c1xx : fatal error C1356: unable to find mspdbcore.dll
12>Compile [x64] Module.VPSettings.cpp [RemoteExecutor: Worker0]: Exited with error code 2 (Force local retry). This action will retry locally
12>Module.VPSettings.cpp
12>c1xx : fatal error C1356: unable to find mspdbcore.dll
12>Compile [x64] Module.MfMediaEditor.cpp [RemoteExecutor: Worker0]: Exited with error code 1353 (Force local retry). This action will retry locally
12>Module.MfMediaEditor.cpp
12>EXEC : cl.exe error : MapViewOfFile failed trying to map 33554432 for C:\Users\chris\Documents\UE_5.6\Engine\Plugins\Media\MfMedia\Intermediate\Build\Win64\x64\UnrealEditor\Development\MfMediaEditor\Module.MfMediaEditor.cpp.obj. ReservedSize: 1073741824 (Error code: 1450)
Note that these weren’t all the errors and the errors aren’t usually consistent, they’re pretty random when it comes to which modules encounter these errors except for MiMalloc.c. Also, I enabled bForcedRetryRemote, so maybe that’s why my build succeeded regardless of these errors
So I was using c5 ec2 instances, which have a 2:1 RAM to vCPU ratio. This was leading to out of memory issues and those random errors referenced in my last reply in the OP. So I upgraded to m7 ec2 instances where the RAM to vCPU ratio is 4:1, and the errors went away (this was with the release branch which at this moment is 5.6.1, not the ue5-main branch). I think this is the minimum ratio needed to avoid issues like these for building Unreal Engine from source with Horde and Uba!
