We have been using Virtual Asset for a while. Since the project’s version control uses SVN, we chosed to share payloads via persistent storage + SMB, with an automated pipeline regularly virtualizing newly added assets in the project and removing payloads that are no longer in use.
However, as the project continues to grow, there are now hundreds of thousands of payload files in remote, which has significantly slowed down the pipeline’s process of checking payload validity and usage. Most of the time is spent traversing the file directory in SMB. We noticed that the payload file structure seems quite similar to DDC, and perhaps we could modify ZenServer to implement similar management functionality? Although ZenServer cannot handle version information, extending a http server seems more feasible than relying on SMB.
I would appreciate some advice on whether we should proceed with this approach or if there are better alternatives. Or if Epic has any planned features that would allow Virtual Assets to be used less relying on Perforce? Thank you!
I’ve followed up with some of the devs working on ZenServer and they said that persistent storage is not a feature that they’d be confident supporting in Zen at the moment.
It’s also worth noting that even if Zen did support that sort of storage and issues like data backups were solved, that you’d still run into the problem that the editor currently only supports 1 DDC graph at the moment and VA uses the DDC api to push data to DDC/Zen/HordeStorage for caching purposes. This feature has been in the backlog for a while but I have had no success in getting it’s priority raised, so you’d still need to implement a custom VA backend in code to communicate to Zen to get around that.
I would appreciate some advice on whether we should proceed with this approach or if there are better alternatives.
Since it sounds like you are only virtualizing content as part of an automated pipeline and it is not being done by the users themselves, have you considered trying to maintain some sort of oplog or data base of files added to you SMB so that you can eliminate the need to traverse the file directory structure?
I have discussed similar ideas in theory with some other licensees who want to be able to run automated passes on their p4 storage to delete unused payload files but find that running p4 commands frequently to get the list of files currently stored there too slow.
If you are letting users virtualize the content as well you might want to consider creating a copy of UE::Virtualization::FFileSystemBackend and add custom code in the push method to write to this oplog/data base to help maintain a record of what has been uploaded. You could then distribute this custom backend as a plugin to avoid editing engine code if you wish.
Or if Epic has any planned features that would allow Virtual Assets to be used less relying on Perforce? Thank you!
Unfortunately there are no plans to expand on the current storage offerings, at least not in the near term.
Sorry for the delayed response due to the holiday. We did implement caching oplog to speed up the search for SMB files, but since the operation of checking file existence and deleting files discretely on SMB is not fast, there are still speed bottlenecks in our automated pipeline. Unfortunately, we will not be switching to p4 for the time being, and we cannot utilize its powerful VFS. In any case, thank you for your answer.