Hello, I have question about Unreal Horde and storage/backends.
We would like to write the same data to 2 backends at the same time (one local, one cloud, data are present in both).
It is possible to specify secondary storage in backend config (so that object is read from the second one if not found in the first one) but that works only for reading out of the box, not writing.
Upon inspecting the code, it seems like the solution could be modifying ChainedObjectStore.cs and propagating the writes also to a secondary backend (ObjectStore).
What could get a bit tricky is properly implementing TryGetWriteRedirectAsync so that everything still works correctly.
Anyone has any experience with this writing data to two (or more) backends?
Hi there Lukas,
Thanks for your inquiry. This doesn’t appear to be a use case we’ve really encountered from others, and not exactly one we have exercised ourselves. Of course you can have separate namespaces backed with different storage backends, but the same namespace with duplicate storages isn’t currently supported. To your point, you could try implementing a new storage backend (IObjectStore, whihc is higher level, or a IStorageBackend which could offer desired redundancy). Are you effectively trying to write a “cache” layer from a storage perspective?
Julian
Hey there Lukas,
Sounds good. Regarding the ETL approach, it’d certainly need to be done by scratch as I don’t believe we have any existing support. Please keep me in the loop with how you fare extending this system as I’ve been starting to coalesce more notes on the extensibility of the system.
Julian
Thanks Julian,
yes, you are right, what we have in mind is basically a cache. Local data that are used when possible and remote (cloud, AWS) data served to remote users.
For a bit more context, we are basically trying to recreate a setup that we used previously where we had our own custom tool downloading data/builds/tools to users (devs). We had the data in office on a shared drive and also in the cloud. People working in the office would get the data from the local shared drive and would download them from cloud only if they were not available locally (basically cache miss). People working from home/remotely would get the data directly from cloud (no need to be connected to VPN, etc.).
So the main goals are saving on the cloud bandwidth for in-office users, making data available to remote users (without the need to use VPN and without data flowing through the office internet connection) and a data backup in case of a failure of a local storage (network drive).
Does it make sense in Unreal ecosystem (Horde, UGS)? I’ll admit that I am not yet familiar with how are artifacts/builds built by Horde downloaded through UGS, so it is possible I am missing something there (e.g. is the UGS downloading data directly, or are data distributed by Horde - something I haven’t looked into yet).
So you think Implenting new storage would be best/easiest way to go about it? As mentioned in the initial post I already considered adjusting ChainedObjectStore.cs but I suspect additional changes might be required to ensure writing works correctly.
Hi Lukas,
I think for this, you’d certainly be able to use a ChainedObjectStore, but to your point the writing aspect will be where you’ll need to provide that duplicate write-over. Thinking aloud, you could try deriving ChainedObjectStore to implement your more nuanced “duplicate write” approach. In doing so, you’d avoid having to diverge the code, and deal with more complex integrations/upgrades down the line. This would be the “safest” bet going forward in my opinion.
There’s also a far less “clever” solutions wherein you just have a primary horde server, and a secondary which is configured to act as a “local cache” (backed by local file store). You could then schedule a regular ETL to “replay” the corresponding namespaces and push them to the local cache. This would keep things “out of source code divergence/modification”.
Regards,
Julian
Hi Julian,
I’ll look into implementing a new and separate version of ChainedObjectStore, as you mentioned that seems to be the safest option.
Having primary and secondary Horde servers is an interesting idea. Out of curiosity, the ETL would need to be implemented manually/from the scratch, correct? There isn’t any support for replication/replay built in?