What’s the expectation for fully remote studios for deploying through UBA?
Should devs working from home also be able to deploy shader compilation to the cloud?
What are the best practices for utilising Horde & UBA when entirely remote?
Thanks
Andrew
With your reference to “the cloud” I assume you are running Horde server and Horde/UBA agents somewhere?
We recommend using a VPN or some kind to avoid exposing Horde and agents on public internet. If you don’t have one today, perhaps something like Tailscale can help. Or if the cloud provider offer a VPN solution. Better support for a “VPN-less” model is on the roadmap but I have no estimates. Good upload and download speeds for people working from home is also recommended. Shaders are less bandwidth intensive than C++ code compilations.
Thanks Carl,
Yes, a Horde server is setup behind a VPN, but not utilising UBA at present. Just curious as to what’s possible for a fully remote studio with regard to deploying compilation/shader jobs to Horde/Agents. In this case we’re running 5.6.1 and I know cloud support works, but was not recommended on 5.4, and 5.5 added support for things like shader compilation.
Is Horde & UBA now stable enough to support fully remote studios, and what are the best practices for setting that up? (any gotchas we need to be aware of).
Thanks
Andrew
Remote shader compilation support is younger and has seen less use internally at Epic, especially in 5.6 so I would not label it stable.
I think it depends on your appetite - how big of an issue are the shader compile times for you?
If you have a VPN that includes developers and Horde server/agents with good connections, it could be worth a shot with the caveat of there still being sharp edges.
I’ve been digging in a bit to try and figure out if it’s even possible to use UBA outside of a direct connection, and I’m not finding the results to be very good, at least with my asynchronous connection, using whatever defaults the system comes with. I can pretty much have 95% of the engine built locally before a single result comes back from the remotes, which makes little sense, so it would seem like I’m doing something horribly wrong.
I recall reading something about there existing a UBA Proxy or some such that is intended to help with situations such as this? But I cannot find any detail about how to operate such a thing if there is.
UBA builds can be very bandwidth intensive, especially with PCH files. Have you tried enabling the UBA visualizer to see where your time is spent? It shows upload and download timing among other things.
There is a relay mode that allows connection to an UBA agent to take different route through a relay rather than direct to an agent’s IP. But it’s still experimental and not officially supported.
The intention is to avoid sending too much traffic through a single VPN gateway (if that’s how you studio operates).
I discovered the UBA_ZONE environment variable setting, while searching around in the forum here, and that sounds like it’s an improvement that might be promising, taking advantage of the zone capability to proxy the files around, am hoping to get a shot to try that out next week. That might be the bit that was talked about around the 5.6 release (i think) to improve UBA usage for remotes, but it is as far as I can tell totally undocumented. It comes automatically if you’re using AWS, so I can see that getting overlooked as something that needs to be mentioned 
Visualizer doesn’t give much insight as to what’s really going on, I don’t think. I probably need to try it with a separate pool with only one or two UBA hosts just to see if it even works to any reasonable degree .. our uba pool right now has around 40 hosts in it, so it’s quite probable that i’m spending the entire build time just trying to feed data to the pool. Also hoping to get a synchronous 2Gbit line soon to solve part of that problem…
But I think the zone proxy feature is probably what I’m looking for to see if that will get it going.
I’d be interested to take a look at this relay bit, too, if you could point me in the direction of the related code, I won’t ask any further support on that
You’re right, UBA_ZONE may reduce traffic needed if set correctly.
Hmm, that’s strange. UBA Visualizer should show if you spend a lot of time uploading or downloading.
As for the relay feature, you would need to set `-UBAHordeConnectionMode=relay` and have a dedicated agent running in relay mode capable of forwarding and modifying nftables on a Linux machine. This setup is complicated and we want to move away from it.
Yes, I would say your upload speed is too limiting for good utilization. At Epic, we recommend developers using UBA from home to have 500 mbit upload.
You can try disabling PCHs to see if that helps, but I suspect it may still not be enough.
well it does say that i’m spending time uploading, but I don’t know what would be out of the norm. On my 50mbps up/1.2gbps down link, with some 40 uba hosts at the other end of the vpn connection, it keeps the uplink saturated for most of the 30 minute build time, it receives only a few files back, and most of the connections drop due to timeouts.
Most of our people are remote and have similarly situated asynchronous up/down situations .. we haven’t really tried it in too many different situations yet.
Setting UBA_ZONE across all the Horde Agent machines does, in fact, make UBA usable by remote. On a 50mbps upload, it only saves a few minutes for a complete build, but on a 300mbps upload, it is capable of getting full support from the entire remote. Without setting the UBA_ZONE, attempting to use UBA from outside of the network was simply unusable for anyone.
Going to try shader compiles next, hopefully that works with lower bandwidth.