I’ve set up a UBA Cache Server and have seen some odd results when storing/retrieving, so I’m interested in gaining a better understanding of how it works.
For my testing, I have 3 machines, one running the UBA Cache Service and 2 others connecting to it. Those 2 machines have project workspaces based off of the same project stream and are synced to the same p4 changelist. Using machine 1 I compiled the Development Editor and I see it populated the cache. I did a full rebuild of Development Editor on machine 1 and could see that it pulled everything from the cache in a fraction of the time. Awesome!
Then using machine 2 (on the same p4 stream and changelist), I compiled the Development Editor. However, it failed to retrieve things from the cache. Based on the UBA cache server logs it seems to be searching for different buckets than what machine 1 used. I attempted to look through the code to find where it calculates and determines the id for the cache bucket, but I’ve so far been unable to pin down where that is. Do you know where I can find that in the UBA code, as well as where it calculates the CAS id for a given file?
Since I couldn’t find that code, I did some experimenting to see if I could figure out the missing pieces. As it turns out, I eventually got machine 2 to pull the files from the cache after doing the following:
Copying the EXACT Visual Studio compiler version from machine 1 to machine 2 (it was very slightly off, 14.38.33144 vs 14.38.33145)
Changing machine 2’s Visual Studio to use the compiler from the same file path that machine 1 did (C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.38.33130 vs C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.38.33130)
Changing machine 2’s workspace to match the same file path that was on machine 1. Machine 1 was at C:\p4 and machine 2 was at E:\p4. I created a Windows junction on machine 2 that linked C:\p4 to E:\p4, and then I opened the project in VS from C:\p4.
It appeared that all 3 things above were needed in order for machine 2 to successfully retrieve the cached files that machine 1 had built and stored. The first one with the compiler version needing to match makes sense, though I was surprised that the full file paths for the compiler and the project workspace needed to match as well. Is that accurate? Or am I maybe misunderstanding something, or maybe failed to configure something properly?
Of course not long after posting this I believe I finally found where it calculates the bucket hash in `UBAExecutor.cs`, and found this handy comment likely explaining things
// Absolute path is set for actions that uses pch.
// And since pch contains absolute paths we unfortunately can't share cache data between machines that have different paths
So given this, I’m curious how Epic handles different workspace paths between things like Horde agents and team members? Even within Horde agents they will have multiple workspaces that would otherwise be able to share cached items if not for the file path. Curious if you use anything that standardizes these file paths, or just generally how you manage things for optimal UBA cache usage?
Ah, hehe. what you should use is “-vfs” (or you can set it in your BuildConfiguration.xml)
What vfs does is that UBT creates virtual paths for everything… platform sdks, sync root, etc… so all response files (.rsp) has “z:\uevfs\Engine\<and-so-on>”.
It also tells UBA how to resolve all those paths… so when the compiler asks for z:\uevfs\Engine\Intermediate\…\Something.rsp, uba will under the hood convert to the right file. So from the compiler’s (and linker’s) perspective everything is on z:\uevfs but we both now they are not
When compile errors etc are outputted, UBA convert the strings too.. so the user see the local file compile/linker errors.
We have been running with this on our farm for a year or so.. I’ve been running with it locally even longer. We are discussing enabling it by default inside epic for everyone.. but there are a couple workflows we don’t fully support yet.. and that is android debugging (the breakpoints doesn’t work).. and there is a bug somewhere in our solution for xcode on mac (I had it working but people have reported it not working).
So with vfs all paths look the same regardless of local layout on your machine and you can then share pch and everything.
The cache can be blazing fast and is very latency tolerant (everything in uba is heavily optimized with latency in mind since so many of us work from home)… I use our cache server from West coast Canada (from my house) and the cache server is located east coast US (~60-70ms latency). here’s a screenshot of LyraEditor built just now for you (The reason I don’t get 100% cache hits is because the machines populating the cache server don’t build LyraEditor but FortniteEditor, plus I have a bunch of local changes).. 1min23sec is not bad given the conditions
I should also mention that I have done a whole bunch really nice optimizations the last two weeks to reduce I/O and networking.. and binaries should be backward compatible.
We have two cache servers for Epic.. one for the farm and one for the devs. The farm cache server serves ~80,000 clients/month and is sending like 25tb/day (and receives 15tb). The churn is just insane since every time someone modifies a header that is included in core pch files everything just rebuilds
The design is quite different from a typical key-value store but that is also what makes it so fast.. not having to run any preprocessors or anything locally like typical other cache system do
Also, I recommend running the cache server on linux if you want to run it for a bunch of people.. mainly because “DeleteFile” is so insanely slow on Windows
oh and yes.. you do need the same compiler version, it is part of the key. It would be possible to change so people can have different compiler versions but I’m pretty sure you would run into trouble if you want to use pch since pch is simply a straight-off memory dump of the compiler state.
The cache system supports something called path hashes… where you essentially can tell the cache system to ignore the hashes of individual files under a path and instead use the supplied hash. If you search in the UBT code for “RegisterPathHash” you can find the places where these are added.. you can ofc hard code a value instead of compiler version and then all of you get same cache keys even with different compiler versions.. but all bets are off when it comes to pch for sure
Awesome, thank you! I was really hoping there was something like this for standardizing the paths. I ran some tests today and this seems to be working really well so far, so I also set this up on some of our Horde agents.
One thing I noticed though is this has impacted the symbol source indexing of builds. I’m guessing the file path references in the symbols are now z:\uevfs, but the source files the p4 records would be referencing would be in the p4 workspace’s local file path syntax, so then that would be what is written as the source index into the PDBs. I’m planning to dig into SymStoreTask.cs to see if there’s a good way to remap the paths to z:\uevfs after getting the p4 depot file mapping, but do let me know if there’s something that already handles this when source indexing symbols and using VFS
Ohh that’s good to know! I’ll have to take a look at that and see if we can cherry pick the latest UBA changes into our engine branch (we’re currently on 5.6).
Also just curious, do you know if these changes are going to get pushed in an update to 5.7? Or are they slated for a later release like 5.8?
Oh yeah, I’d definitely be asking for trouble if I tried to mix and match compiler versions haha. So it totally makes sense that it would result in a different hash.
Though I was a bit more surprised that the same exact compiler version, only living in a different file path, also resulted in a cache miss. Though I’ll take a look to see if using “RegisterPathHash” might be able to account for this while still being able to match only exact compiler versions. Thank you for all of the info here, it’s greatly appreciated!
What is SymStoreTask? I have not heard of this before.. maybe something I need to fix then.
I know there likely are a couple of bugs in the code that needs fixing. I think clicking ”go to source” in the editor might not work.. I have a change that I have meant to submit for a long time now.. will get it in.
hmm, maybe pathhashes were not implemented in 5.6… in that case it will hash the binaries and if cl.exe is different you will not get a cache hit. it is a bit annoying that microsoft can patch the exe straight in the install so even if you think you have the same version you might not
SymStoreTask is a Build Graph task that handles symbol storing (and source indexing) of a build’s symbols. The source indexing is essentially creating a mapping between the source files local file path and its location/revision in source control (p4 depot). This way when you open a crash dump it can display the source code for the exact revision the crash occurred on.
So what seems to be happening here is because the compiler/linker believes the source files local path to be z:\uevfs\root\… then that is what it is looking for in the mapping. However, when this task is building the source mapping, it’s using the actual file path, E.g. c:\p4\project\…
Early indications show that the following quick and dirty fix I did today seems to be working. I added the following in WinPlatform.Automation.cs “AddSourceIndexToSymbols”, under where its writing the original mapping:
Writer.WriteLine("{0}*{1}*{2}", SourceFile.FullName, SourceInfo.DepotFile.Replace("//", ""), SourceInfo.Revision);to now be:
That’s awesome! Another question came to mind, does UBA have a mechanism for caching things locally, and utilizing that local cache before checking the UBA cache server? So for example, if I do a full build and it pulls everything from the remote cache server, is there a way for it to cache that locally so that if I do another full build (assuming same files) it will quickly pull them locally instead of having to fetch them again?
