UnrealBuildTool bug (incorrect build configuration data caching implementation; lead to build fail)

Development Programming & Scripting
1

Hello,

There is a clear (easily detectable via code analysis) bug in UnrealBuildTool tool which lead to build fail under some (rather random) circumstances with the following message (for Windows platform, though another platforms may be affected as well):

error MSB3075: The command “<…>\Build\BatchFiles\Build.bat <…> -waitmutex” exited with code 5. Please verify that you have sufficient rights to run this command.

Please find detailed report below:

Build type : Source

Version : 4.16.2-release / Windows / Visual Studio 2015

Detailed description of the issue:

Under some circumstances source build of UE4 (on Windows platform) failed with one or multiple (random number) of the following messages:

error MSB3075: The command “<…>\Build\BatchFiles\Build.bat <…> -waitmutex” exited with code 5. Please verify that you have sufficient rights to run this command.

where <…> - placeholders for parts of message which are specific for particular case (depend on path to project folder, build configuration, etc.)

The build log also contains following related unhandled exception stacktrace per one such message:

UnrealBuildTool Exception: System.IO.IOException: The process cannot access the file '<UE-Source-Build-Folder>\Engine\Intermediate\Build\XmlConfigCache.bin' because it is being used by another process.  
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)  
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)  
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)  
at System.IO.File.Open(String path, FileMode mode, FileAccess access, FileShare share)  
at UnrealBuildTool.XmlConfigData.Write(FileReference Location) in <UE-Source-Build-Folder>\Engine\Source\Programs\UnrealBuildTool\System\XmlConfigData.cs:line 124  
at UnrealBuildTool.XmlConfig.ReadConfigFiles() in <UE-Source-Build-Folder>\Engine\Source\Programs\UnrealBuildTool\System\XmlConfig.cs:line 93  
at UnrealBuildTool.UnrealBuildTool.GuardedMain(String[] Arguments) in <UE-Source-Build-Folder>\Engine\Source\Programs\UnrealBuildTool\System\UnrealBuildTool.cs:line 402  
at UnrealBuildTool.UnrealBuildTool.Main(String[] Arguments) in <UE-Source-Build-Folder>\Engine\Source\Programs\UnrealBuildTool\System\UnrealBuildTool.cs:line 927

This stacktrace is self-explanatory and provide the key to the root of issue: there is an interprocess race condition inside the UnrealBuildTool implementation, and in particular - creation of build configuration data cache file named XmlConfigCache.bin. During build of Unreal Engine with Visual Studio, the number of simultaneous UnrealBuildTool processes are spawned and there is race condition occur when each of that processes trying to create XmlConfigCache.bin if it is not exist. Depending on timing one or more of that processes may fail. It is also possible that none of them will fail under lucky timing.

IMPORTANT: because one UnrealBuildTool process will always success and will create the XmlConfigCache.bin, the issue arise only during very first (fresh) build. All consecutive builds will just use already existing XmlConfigCache.bin file and issue will not take the place (Clean command doesn’t delete this file). This should be taken into account during reproducing attempts (see below).

While description above is made for Windows platform, it seems that same issue may arise on other platforms if they also utilize parallel build option (had no chance to test).

Repro Steps:

Because it is race condition (and moreover IPC one), it hard to provide guaranteed repro steps, but below are ones which should provide better chances to reproduce the issue with minimal efforts:

  1. Make fresh UE build from source
  2. If issue is not arise, delete <UE-Source-Build-Folder>\Engine\Intermediate\Build\XmlConfigCache.bin file
  3. Run regular build again (not Rebuild !). Because build is already done, and there are nothing to build, it will be just dry-run for the issue (UnrealBuildTool processes will be spawned as required, but no actual compiling/linking task will be performed, so it will be quick)
  4. Repeat from step 2 until issue arise (though, there are no guarantee; while in my case I got issue almost each attempt)

Instead of making fresh UE build for step 1 above, existing build may be used - just start from step 2.

2

I’m also running into this problem. In my case it happens when I do a fresh UE build from source. It’s building on a virtual machine with 32 cpu cores and 32 GB RAM. I noticed it happening less when I limit the maximum number of concurrent C++ compilations in Visual Studio to something like 8, but it still seems to happen quite a lot.

3

We are also running into this issue. Our issue arises when we try to limit the Maximum count of Processors the VM can use. If we set it, we get this crash at a semi-reliable rate. If we don’t it goes through but fails when tried to create the .pdb of the project (Project-Name.pdb)

4

This issue is still plaguing us. Has Epic even seen this post?

Does anyone know if this has been fixed in UE 4.17+?

5

We’re also still seeing this issue in UE 4.17. Haven’t tried 4.18 yet though.

6

I’m having this issue as well. Tried both 4.17 and 4.18 on both Visual Studio 2015 and 2017. Sometimes it takes over ten tries before I can get a successful build. It’d be nice if an Epic staff member could comment on this issue.

7

I created a Pull Request to work around this race condition.

https://github.com/EpicGames/UnrealEngine/pull/4191

Code catches this specific error and will retry up to five times with a small random sleep to give the conflicting process a chance to finish.

Here’s a Patch file for those who don’t have/want Github Access. (Had to give it a .txt extension because UDN doesn’t trust the .patch extension.)

8

Should be fixed in 4.18 via https://github.com/EpicGames/UnrealEngine/commit/0fb9287f0a546f997fd9ea9e7339f3f7e9dc1afc – so the above is only needed in 4.16 & 4.17.

9

@Philippe23: I am looking at integrating this fix into our Unreal Build but I am stumped at how rigid the UBT is. Is there anything specific that needs to be done to add a file to the UBT setup phase? Because I can manually add the required new file into the VS project to compile but it’s not getting added automatically.

10

@Volvary: Not clear on which version fix you’re trying to apply / which UE4 version you’re trying to patch.

I’ve just been modifying .../Engine/Source/Programs/UnrealBuildTool/System/XmlConfig.cs b/Engine/Source/Programs/UnrealBuildTool/System/XmlConfig.cs in place for my changes. I’m still running 4.16.2.

I haven’t tried the 4.18 fix – figure I’ll get it when we make our way to 4.18.

11

I fixed it by pushing the missing class into the file Ben Marsh modified in the commit you sent.