We have done some small tests with UBA Cache Server and it seems promising.
We would like to to setup a server that could be used in production shared amongst multiple projects and would like to know what are the hardware recommendations.
One main question is how much disk space is needed.
We do not have anything in cloud, just a single office network.
[Attachment Removed]
Hey,
As for compute, the server uses more or less zero compute except when it is doing maintenance. When doing maintenance it goes as wide as it can to try to finish as quickly as possible. While maintenance is going it can’t handle any requests. The only reason you want more than say 4 cores is maintenance.
For storage, this mostly actually comes down to how often your core headers are modified, and especially if you use pch, how often they recompile (since they are often really big). Inside the cache server everything is compressed and backed by cas storage so files with same content is only stored once. I don’t know how many devs you have submitting code and how many branches/games you have and it all depends on these variables.
What I can share, is Epic’s setup… which is quite insane and most likely nowhere near what you need.
The machine running the cache is a i3en.12xlarge aws instance with 27TB storage. This cache hosts our entire build farm and covers maybe ~7 branches that are very different. All branches build win64,mac,linux,ios,android,xsx,xb1,ps4/5,switch1/2 and probably something more I don’t remember. The cache is running on linux and host both win32/linux/mac clients.
The churn is very high and there are often 20+ active clients connected to the cache at any point in time during the week. Maintenance only runs when there are zero connections or when it is running out of disk space, which means it usually only manages to run maintenance a few times during off-hours.. and a maintenance run takes between 3-30 seconds depending on how long time it was since last maintenance. Running out of space means that it starts removing cas files starting with oldest accessed first.
Here’s stats from last friday (less busy day) taken directly from the log
UbaCacheServer - Stats since boot (46d5h38m ago)
UbaCacheServer - CacheServer 444.8m hits, 220.7m misses
UbaCacheServer - StorageServer cas 1.0g (568.0tb) sent, 132.9m (358.9tb) received
UbaCacheServer - Stats since last maintenance (18h15m7s ago)
UbaCacheServer - Peak connection count: 60
UbaCacheServer - Maintenance starting after 6894020 added cache entries
UbaCacheServer - Removed 312.2k cas entries (marked as not existing)
UbaCacheServer - Found 15.2m (27.8tb) cas files and 231.0k deleted by overflow (3.4s)
Since boot two months ago (this includes Epic’s christmas break) it has sent 1 billion cas entries (files) at 568TB and received 132 million entries. It peaked at 60 connections simultaneously during the last 18 hours.. and also during those last 18 hours it added 6 million new cache entries. That is 92 new cache entries added per second if evenly spread out over those 18 hours :-).. one cache entry is one build action (mostly compiling one file or link one binary)
It is hard to recommend cache size. If I were you I would start with 1-2TB and see how much history it can hold.. it so much depends on your number of different targets (platform/debug/development/shipping etc) and how often someone modifies headers that affects tons of compile actions. and ofc also if you are using pch or not
[Attachment Removed]
Hehe, you want it infinitely fast 
[Attachment Removed]
I’ve been running for several months with “-expiration=1209600 -capacity=1024” (2 week expiration, 1Tb capacity) and it seems to be fine, just did a quick restart on it while no one was using it to grab the current status:
```
UbaCacheServer - Maintenance done! (19ms) CasFiles: 83.5k (386.0gb) Buckets: 6 Entries: 55.0k Oldest: 77d14h0m LongestUnused: 13d23h21m MaxMaintenanceMem: 0b/134.2mb
```
What do you use for parameters for it?
fwiw, we also run our Zen cache on this, and it’s a Epyc 7402 24-core around 3ghz, with quad 3.2Tb NVMe disks striped together for a 12Tb volume. It’s plenty plenty fast as a file server, that’s for sure. I don’t think it’s ever been put over 10% CPU usage in the year it’s been in service, nor have i seen the disks bottleneck anything at all. This may even be overkill, but really, can your cache ever respond too fast?
[Attachment Removed]