UnrealBuildAccelerator issues with link and cache

Development Pipeline & Plugins
, ,
1

Hello,

I’m trying out UBA on 5.5.

I’ve setup a test server which also runs UbaCache.exe, and two workers.

So far, distribution and cache seem to work and I’m pleased I did not have big issues. Thanks for the existing documentation pages, even if they are not complete obviously, they are already really helpful.

I have two issues that I don’t know if they are from something on our end or not.

  1. Link error LNK1168

When UBT needs to rebuild some modules, for example just by touching a source.cpp file, it ends up with systematic link errors such as this one:

LINK : fatal error LNK1168: cannot open C:\dne\monorepo-preview\Game\TEST\Binaries\Win64\UnrealEditor-TEST.dll for writing

The files exist but do not appear to be locked.

Cleaning and then doing a full build from start does work and builds a working editor.

2.Cache root path issues

During build, caching works except for our in-house plugins and games. I suspect this is because we patched the engine to store them outside of UE’s root directory.

This happens with a lot of logs about PATH/FILE WITHOUT ROOT:

UbaCacheClient - PATH WITHOUT ROOT: C:\dne\monorepo-preview\DNE\Plugins\GameSystem\DNEDialogue\Source\DNEGenDialogueEditor\Public\DN (inside C:\dne\monorepo-preview\Game\TEST\Intermediate\Build\Win64\x64\UnrealEditor\Development\TESTEditor\TESTEditorModule.cpp.d at offset 147677)

UbaCacheClient - PATH WITHOUT ROOT: C:\dne\monorepo-preview\DNE\Plugins\Core\DNECondition\Intermediate\Build\Win64\UnrealEditor\Inc\ (inside C:\dne\monorepo-preview\Game\TEST\Intermediate\Build\Win64\x64\UnrealEditor\Development\TESTEditor\TESTEditor.Shared.rsp at offset 17)

UbaCacheClient - FILE WITHOUT ROOT: C:\dne\monorepo-preview\DNE\Plugins\GameSystem\DNEDialogue\Source\DNEGenDialogueEditor\Public\DNEAssetTypeActionDialogue.h (TESTEditorModule.cpp (Compile [x64]))

UbaCacheClient - PATH WITHOUT ROOT: C:\dne\monorepo-preview\DNE\Plugins\Core\DNECondition\Intermediate\Build\Win64\x64\UnrealEditor\ (inside C:\dne\monorepo-preview\Game\TEST\Intermediate\Build\Win64\x64\UnrealEditor\Development\TESTEditor\UnrealEditor-TESTEditor.dll.rsp at offset 1603)

and also happens inside "CmdKey"s or .lib files.

When looking at the related files I can see it points to full paths, is there something I can configure anywhere to avoid UBA getting confused?

Or would you say our patch introduces full paths, but the system expects only relative ones?

I understand this has to do with sharing cache artifacts between workspaces in different locations, but I saw that VFS was to solve this in 5.6.

2

Hey there Antoine,

I’d think that this is likely related to code existing outside of the root. VFS could probably address this, but I’m not confident. Do you happen to have some of your patches to the engine handy for me to review?

A couple of related, but minor questions:

  • Can you provide your BuildCofniguration.xml and command line?
  • Can you provide the initiating machines UBT log?
  • Does this work without caching?

Kind regards,

Julian

3

Hi Julian,

I’ll get you the patch and logs asap.

In the mean time, I can confirm that the link issue is present even without caching. However it only occurs on our plugins/games dlls, also related to our patch I guess!

[15/20] Link [x64] UnrealEditor-ALFEditor.dll

LINK : fatal error LNK1168: cannot open C:\dne\monorepo-preview\Game\TEST\Binaries\Win64\UnrealEditor-TESTEditor.dll for writing

Link [x64] UnrealEditor-TESTEditor.dll: Exited with error code 1168 . The build will fail.

Link [x64] UnrealEditor-TESTEditor.dll: WorkingDirectory C:\dne\monorepo-preview\UE\Engine\Source

Link [x64] UnrealEditor-TESTEditor.dll: C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.38.33130\bin\Hostx64\x64\link.exe @“C:\dne\monorepo-preview\Game\TEST\Intermediate\Build\Win64\x64\UnrealEditor\Development\TESTEditor\UnrealEditor-TESTEditor.dll.rsp”

The .rsp for failing link steps specify the .dll file with full path:

/OUT:“C:\dne\monorepo-preview\Game\TEST\Binaries\Win64\UnrealEditor-TESTEditor.dll”

4

Hey Antoine,

And one final question, does it work when not using UBA? Quite related documentation around debugging UBA - try -NoUBA or -NoUBALocal

Whilst this is likely related to your patch - we do have an unwritten policy where we want UBA to reflect the local workflow; so if it works in local we should see what’s going on in the UBA case.

Regards,

Julian

5

Hello Julian,

Yes, it does work without UBA, wether with the default parallel executor or with our internal FASTBuild executor.

I didn’t make it clear, but the patch I mentioned has been in use for several years now and we shipped multiple projects with it. We have plans to drop it in the future and rely on Unreal’s native ‘AdditionalDirectories’ feature.

Edit : see this question for example [Content removed]

6

Hey Antoine,

Thanks for the details here.

I’d really recommend rebuilding UBA under debug config, and collecting the related logs here so that I can send this off to one of the subject matter experts in this area.

Julian

7

Here’s the log from UBT after I `touch` a .cpp file, and do a standard build.

I’ve previously compiled all UBA projects in Debug from the Uba solution (afaik the build already places them in the correct folder).

You can see a linker error that can’t open the DLL. However the file exists.

I tried running UbaCli.exe under VS to debug what’s happening, and it also fails in the same way. My command line:

UbaCli.exe -workdir=C:\dne\monorepo-preview\UE\Engine\Source -cache=“192.168.3.127” local “C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.38.33130\bin\Hostx64\x64\link.exe” @C:\dne\monorepo-preview\DNE\Plugins\Core\DNEAnimationCore\Intermediate\Build\Win64\x64\UnrealEditor\Development\DNEAnimationCoreRuntime\UnrealEditor-DNEAnimationCoreRuntime.dll.rsp

However, if I run the link step myself from a shell, it works.

So I’m thinking, but this is a wild guess, that Detour might have something to do with this? I’ve also noticed that we had several include directives with a trailing space in the filename, such as

include "SomeHeader.h "

that worked without anyone noticing for years, on msvc and clang-cl, but failed instantly under UBA with a “file not found” error.

I’ve also attached our patch, named DNEGameSupport. It’s quite pervasive, sorry about that.

8

Hey Antoine,

Unfortunately I don’t see the patch attached (just the logs - and thanks for that).

>You can see a linker error that can’t open the DLL. However the file exists.

  • Good to know; [mention removed]​ do we have any thoughts on this? So it seems like there are some patches (aforementioned - still need to obtain the code), but it sounds like we have divergences that allow us to work outside of the project root (this is similar to AdditionalDirectories), so I expect the issue is coming from this.

It’s a stretch, but if you fix the spaces, does link start working? I’m curious why I’m not seeing the file not found error with respect to the headers in the logs.

Kind regards,

Julian

9

I fixed the include directives already, they were not related to these link issues.

The patch should have been in my zip file, in the form of DNEGameSupport.diff, I’m adding it to this message anyway.

10

>I fixed the include directives already, they were not related to these link issues.

Good stuff - thanks for clarifying.

And apologies Antoine - I missed that.

I’ll see if I can reproduce this locally.

Kind regards,

Julian

11

Hey there Antoine,

Just a heads up that I’m still working to reproduce this locally. I’ll let you know how it goes, as I’ve decided to step back and try to get it working through the AdditionalPluginDirectories route first, as opposed to trying to reimplement local divergences.

Kind regards,

Julian

12

Hey there Antoine,

Thank you very much for your patience - I’ve been pretty backed up with other work the last couple days due to internal conferences.

So I’ve tried at great length to reproduce this with the AdditionalPluginDirectories, and I have yet to reproduce this as such (I’ve migrated over 100+ plugins to various external folders - mapped via uproject AdditionalPluginDirectories). After reviewing the breadth of the divergence, I can only assume that this issue is likely emanating from such changes. UBT and UBA do have some coupling together. That being said, I would still expect a raw UBACli.exe command to work here. Would it be possible for you to run your command above:

UbaCli.exe -workdir=C:\dne\monorepo-preview\UE\Engine\Source -cache="192.168.3.127" local "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.38.33130\bin\Hostx64\x64\link.exe" @C:\dne\monorepo-preview\DNE\Plugins\Core\DNEAnimationCore\Intermediate\Build\Win64\x64\UnrealEditor\Development\DNEAnimationCoreRuntime\UnrealEditor-DNEAnimationCoreRuntime.dll.rsp

… with debug UBA binaries (as described here). This should at least help with verbosity. This could be somewhat related to the files being pulled in from out of sync root - but again I’d expect detours to help with that (unless we are missing some type of virtualization/map update [mention removed]​ ).

13

Hi Julian,

Thanks for taking the time.

It’s good to know that AdditionalPluginDirectories work correctly, it’s one more reason for us to migrate towards this. However to be clear we are _not_ using it yet, as our patch does kind of the same thing.

I’ve compiled all UBA project in debug from UBA.sln, and get the same error when running what you asked.

`PS C:\dne\monorepo-preview\UE\Engine\Binaries\Win64\UnrealBuildAccelerator\x64> ./UbaCli.exe -workdir=C:\dne\monorepo-preview\UE\Engine\Source -cache=“192.168.3.127” local “C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.38.33130\bin\Hostx64\x64\link.exe” @C:\dne\monorepo-preview\DNE\Plugins\Core\DNEAnimationCore\Intermediate\Build\Win64\x64\UnrealEditor\Development\DNEAnimationCoreRuntime\UnrealEditor-DNEAnimationCoreRuntime.dll.rsp
UbaCli v5.5.4-Uba_v1.0.0-0 (DEBUG) (Rootdir: “C:\ProgramData\Epic\UbaCli”, StoreCapacity: 20Gb)

UbaServer - Created in DEBUG
---- Starting trace: Debug ----
UbaClient (0aada7a7-cca7-481f-bde8-b063340818b3) - Connected to server… (0x0000048B94070290)
UbaClient (0aada7a7-cca7-481f-bde8-b063340818b3) - Connected to 192.168.3.127:17157 (f269faa9-ec91-4ad1-948d-d0817cb91664)
UbaStorageServer - Database loaded from C:\ProgramData\Epic\UbaCli\cas\casdb (v32) in <1ms (contained 0 entries estimated to 0b)
UbaCacheClient - PATH WITHOUT ROOT: (inside CmdKey C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.38.33130\bin\Hostx64\x64\link.exe at offset 0)
Running C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.38.33130\bin\Hostx64\x64\link.exe @C:\dne\monorepo-preview\DNE\Plugins\Core\DNEAnimationCore\Intermediate\Build\Win64\x64\UnrealEditor\Development\DNEAnimationCoreRuntime\UnrealEditor-DNEAnimationCoreRuntime.dll.rsp
LINK : fatal error LNK1168: cannot open C:\dne\monorepo-preview\DNE\Plugins\Core\DNEAnimationCore\Binaries\Win64\UnrealEditor-DNEAnimationCoreRuntime.dll for writing
Error exit code: 1168
UbaClient (0aada7a7-cca7-481f-bde8-b063340818b3) - Disconnected from server… (0x0000048B94070290) (0)`(had to run it from a VS Powershell prompt, otherwise linker fails because it’s not able to find mt.exe)

14

replying to avoid automatic closing

15

Hey Antoine,

A couple of follow up suggestions to see if we can further narrow things down *from the divergence*.

Can you try:

  • Getting the new binaries for UBA from ue5-main (they should be backwards compatible)
  • Try running with uba (no cache) against BlankProgram
  • In the failure case please provide the session logs
  • Are you utilizing any kind of symlink?

Julian

16

Hi Julian,

I won’t be able to pursue this for two weeks but I’ll be coming back at it. I hope we can avoid closing the ticket

17

Hey Antoine,

Sounds good - I’ll keep it parked in “open”.

Kind regards,

Julian

18

Hi Julian,

Here’s what I did:

  • checkout branch 5.6 from Unreal’s github (ue5-main fails in GitDependencies due to invalid compression)
  • generate Uba.sln and compile it in Debug, copy the binaries to my test bed
  • run Build.bat BlankProgram Development Win64
    • the full build completes successfully
  • touch BlankProgram.cpp to trigger a recompile
  • run Build.bat BlankProgram Development Win64 -UbaLog
    • the build fails with the infamous LINK : fatal error LNK1168: cannot open ..\Binaries\Win64\BlankProgram.exe for writing

See the attached log, which mentions the error.

We don’t use symlinks in our source setup.

19

Hi Antoine,

Julian forwarded these things to me and I must say it looks very strange. What version of windows are you on? I can see that the code in uba potentially is wrong but I don’t know how to reproduce this. Do you have special command line options when building BlankProgram?

For more details… this is what is weird in your log file. It calls CreateFileW on BlankProgram.exe with “read”.. and succeeds.. and then it closes it and opens it again with “read”. This is what cause the “ALREADYEXISTS” weird error.. which I will look into how that can happen.

D CreateFileW ..\Binaries\Win64\BlankProgram.exe RPC_MESSAGE CreateFile T NtCreateFile 300004 (C:\dne\monorepo-preview\UE\Engine\Binaries\Win64\BlankProgram.exe) -> Success D GetFileInformationByHandle (file) 300004 (#) -> Success (size: 16588800) D NtClose 300004 (#) -> Success D CreateFileW ..\Binaries\Win64\BlankProgram.exe D NtCreateFile ALREADYEXISTS (C:\dne\monorepo-preview\UE\Engine\Binaries\Win64\BlankProgram.exe) -> Error

But also, the question is why it opens it for read multiple times in your log but not in mine… in my build (same sdk and everything) it opens the file for write straight away.. here’s the same place in the log on my machine

D CreateFileW ..\Binaries\Win64\BlankProgram.exe T NtClose 512 (UNKNOWN) -> Success D NtCreateFile (MEMORY) WRITE 300003 (E:\dev\fn\Engine\Binaries\Win64\BlankProgram.exe) () -> Success D CreateFileMappingW (MEMORYFILE) File 300003 Protect 4 Size 2097152 (E:\dev\fn\Engine\Binaries\Win64\BlankProgram.exe) -> 300004 D MapViewOfFileEx 300004 Size 2097152 (E:\dev\fn\Engine\Binaries\Win64\BlankProgram.exe) -> 0x20b55df0000 T NtClose 300004 (E:\dev\fn\Engine\Binaries\Win64\BlankProgram.exe) -> SuccessI will see if I can repro the bug on my machine with a unit test and come back to you here.

20

I’ve investigated further and only explanation I can see is that this is a CreateFileW with GENERIC_WRITE and OPEN_EXISTING and there is an unimplemented path in uba for that in combination with “output files”.. “.exe” is an tracked as an output file from the linker.

I still don’t understand why you get this while I have never seen this with the 100s of devs using this daily.. but I will add code to cover this case. Only thing that sucks is that when you link remotely your path will need to download the old .exe file just to then overwrite it :slight_smile: