Using UBA to distribute custom workloads

[anonymous-edc (Anonymous User)]

Hi,

I’d like to know if, and how, I could use UBA to distribute other workloads than compilation. As I understand, the detouring architecture should allow this even if UE only uses it for engine and shader compilation at the moment.

My current goal is to distribute clang-tidy analysis, which is tedious to integrate natively to UBT (for reasons pertaining to both UBT and clang-tidy itself, mostly argument passing and rsp shenanigans) and I’d have an easier time treating UBT as a black box, getting the compile_commands.json and running clang-tidy myself with UBA as a wrapper.

From what I understand, UbaCli.exe can be used to that end, but I’m missing some instructions on how to configure it. What i’d like to run is:

UbaCli.exe <config arguments> -visualizer remote clang-tidy.exe <tidy arguments>

I suspect I need to use -coordinator=Horde or something, but UbaCoordinatorHorde does not compile.

Can I get this to work?

[Attachment Removed]

[Honk1 (Honk)]

Hi Antoine,

Yes, you are right in that UBA is a fairly generic file system virtualization and it might be fine to use clang-tidy distributed. I’m actually in the middle of a refactoring to make it more robust running remotely (The detoured NtCreateFile is horrendous).

The best way to test if something can run remotely out of the box is to use UbaCli as you have figured out. Here’s the command lines I’m using when doing development:

UbaCli.exe -workdir=<workdir> native <application> <commandline>

Workdir is the working directory for the application. “native” is the mode and does no detouring, it should always work.. if it doesn’t, then it is likely something is wrong with command line or something… application is obviously the application followed by its command lines.

Once this is working you can replace “native” with “local”. Local is making the application go through detouring.. this is the first step towards running remotely and do many of the optimizations to minimize kernel calls.

Next step is “agent”. agent is a mode where UbaCli spawns an UbaAgent internally in-proc. It should be very similar to running a separate agent but is a good way to iterate on issues etc.. being able to put breakpoints etc.

And last step is “remote”. Remote means that UbaCli waits for an UbaAgent to connect to it. You can test this mode locally by just spawn “UbaAgent.exe -host=localhost” on the same machine and it should work. After that you can take UbaAgent.exe and copy it to another machine to test so it works remotely.

ok, say all this is working, now you want to run multiple commands. This is where you can use a yaml file. Instead of having an <application> in the command line you can have a <.yaml> file and in that file you have the list of commands and their dependencies. You also want to change back to “local” mode with the yaml file to let the local machine help out.

Look at the bottom of UbaScheduler.h for syntax… or run UBT with -UBAActionsOutputFile=SomeFile.yaml to learn how the format is.

If you want to use horde as a coordinator you can as you say compile UbaCoordinatorHorde and use that. By default it should use all the configs you’ve setup in the unreal .ini files (It will not read BuildConfiguration.xml like UBT). This path is quite experimental since I’ve mainly used it to be able to run tsan and asan on uba (they don’t like dotnet applications). Once you have compiled UbaCoordinatorHorde you add -coordinator=Horde before the mode parameter

Hopefully you can get it to work Good luck!

[Attachment Removed]

[Honk1 (Honk)]

It might be that there are moments when UbaCoordinatorHorde.cpp code is outdated and wrong since it is not a tested path on our farm. It might be that you need to look that file and see if there are fixes in later versions that makes it compatible with UbaHordeAgentManager and friends.

[Attachment Removed]

[Honk1 (Honk)]

That sounds strange. We have all our horde uba related stuff in BaseEngine.ini and when I run UbaCli using UbaCooordinatorHorde it automatically picks up those settings (including oidc, I think at least since it works). Unfortunately I don’t know super much about unreal and .ini files to be able to guess what is going wrong for you.

Re: command line, maybe we should make it possible to provide command line to the coordinator somehow. Maybe -coordinator=“Horde <arguments” or something?

Nice to hear it works though. Keep me updated on how it goes

[Attachment Removed]

[Honk1 (Honk)]

How are things going?

[Attachment Removed]

[anonymous-edc (Anonymous User)]

Hi Henrik,

I’ve got it to work and even with few agents, our analysis time went down from 1 day to about 1 hour.

I only had to translate the compile_commands.json emitted by UBT to a .yaml file that UbaCli understands (its yaml parser seemed a bit picky and it worked without issue.

Let me add that we’d be very interested by built-in support for clang-tidy in UBT, similar to how PVS Studio is integrated.

Thanks for your help.

[Attachment Removed]

[Honk1 (Honk)]

Great that you got it working yes, the yaml parser is best-effort lightweight tiny thing for me to be able to run asan and tsan on linux on the uba code without UBT (asan/tsan do not like dotnet processes).

the yaml path might miss a bunch of features but good enough for simple things.

I will forward the clang-tidy support in UBT to people here.. don’t know how hard it would be to add.

[Attachment Removed]

[anonymous-edc (Anonymous User)]

Hi Henrik, thanks for the details.

I’ll try to get UbaCoordinatorHorde to build, at the moment it emits compilation errors but I didn’t look into it yet to see if I can patch it easily (see attached log).

Is it correct that UbaCli.exe looks for coordinator DLLs in its own working directory, implying that it should not be called from another directory?

C:\dne\monorepo-preview\UE\Engine\Source\Developer\UbaCoordinatorHorde\Private\UbaCoordinatorHorde.cpp(42,12): error C2664: 'void FUbaHordeAgentManager::SetAddClientCallback(FUbaHordeAgentManager::FAddClientCallback (__cdecl *),void *)': cannot convert argument 1 from 'uba::Coordinator::AddClientCallback (__cdecl *)' to 'FUbaHordeAgentManager::FAddClientCallback (__cdecl *)'
                m_manager.SetAddClientCallback(callback, userData);
                         ^
C:\dne\monorepo-preview\UE\Engine\Source\Developer\UbaCoordinatorHorde\Private\UbaCoordinatorHorde.cpp(42,34): note: This conversion requires a reinterpret_cast, a C-style cast or parenthesized function-style cast
                m_manager.SetAddClientCallback(callback, userData);
                                               ^
C:\dne\monorepo-preview\UE\Engine\Source\Developer\UbaCoordinatorHorde\Public\UbaHordeAgentManager.h(23,8): note: see declaration of 'FUbaHordeAgentManager::SetAddClientCallback'
        UBACOORDINATORHORDE_API void SetAddClientCallback(FAddClientCallback* Callback, void* UserData);
              ^
C:\dne\monorepo-preview\UE\Engine\Source\Developer\UbaCoordinatorHorde\Private\UbaCoordinatorHorde.cpp(42,12): note: while trying to match the argument list '(uba::Coordinator::AddClientCallback (__cdecl *), void *)'
                m_manager.SetAddClientCallback(callback, userData);
                         ^
C:\dne\monorepo-preview\UE\Engine\Source\Developer\UbaCoordinatorHorde\Private\UbaCoordinatorHorde.cpp(82,5): error C2039: 'SetPool': is not a member of 'FUbaHordeAgentManager'
                m.SetPool(info.pool);
                  ^
C:\dne\monorepo-preview\UE\Engine\Source\Developer\UbaCoordinatorHorde\Public\UbaHordeAgentManager.h(14,7): note: see declaration of 'FUbaHordeAgentManager'
class FUbaHordeAgentManager
      ^
C:\dne\monorepo-preview\UE\Engine\Source\Developer\UbaCoordinatorHorde\Private\UbaCoordinatorHorde.cpp(83,5): error C2039: 'SetMaxCoreCount': is not a member of 'FUbaHordeAgentManager'
                m.SetMaxCoreCount(info.maxCoreCount);
                  ^
C:\dne\monorepo-preview\UE\Engine\Source\Developer\UbaCoordinatorHorde\Public\UbaHordeAgentManager.h(14,7): note: see declaration of 'FUbaHordeAgentManager'
class FUbaHordeAgentManager
      ^

[Attachment Removed]

[anonymous-edc (Anonymous User)]

I’m trying to get it to work with the smallest patch possible.

I’ll be documenting here the problems I faced:

• In UbaCli.exe context, GConfig hierarchy only knows about .ini files in Engine/Programs/Saved/UbaControllerHorde, so I can’t reuse my game/engine configuration and it seems brittle to add some config inside a Saved directory
• UbaController settings can be set via a BuildConfiguration.xml, but this configuration source only allows disabling the provider, so FUbaHordeConfigParser cannot be instructed that it’s enabled even though all the needed configuration is there
• FUbaHordeConfig seemingly supports getting configuration from FCommandLine, but this is blocked by uba::WrappedMain that errors on unknown argument
• UbaCli.exe doesn’t compile either on 5.6
• After temporarily copying configuration to Programs/UbaCoordinatorHorde/Saved/Config/Windows/Engine.ini, UbaHorde fails to connect to the Horde server because the url must again be configured somewhere else (UE_HORDE_SERVER env var, or [Horde]ServerUrl in .ini config)
• OidcToken.exe must be made accessible via PATH or copied in the workdir of UbaCli.exe

And then I was finally able to reach Horde and have my task ran on an agent! I’ll see how I can get all that to work in our setup.

[Attachment Removed]