I downloaded the Slay Animation project for the test. So I have the asset hydrated to virtualize via UnrealVirtualizationTool.
I set up the Perforce Source Control and the Shared Derived Data Cache on the server. I’ve created a payload_metainfo.txt without knowing its purpose. And I uploaded the assets to the depot.
But I don’t understand why the Virtualization Tool don’t virtualize the assets. It seems that asset payloads are missing, but I don’t figure what is a payload and how to get them.
What is a payload? How are they generated ? Why does it seem to be needed for virtualization ?
Here's the log I get from the UnrealVirtualizationTool :
LogVirtualizationTool: Display: Initializing...
LogVirtualizationTool: Display: Logging process to '********/UnrealEngine/Engine/Programs/UnrealVirtualizationTool/Saved/Logs/UnrealVirtualizationTool.log'
LogVirtualizationTool: Display: Cmdline parameter '-Verbose' found, no longer supressing Display log messages!
LogVirtualizationTool: Display: Cmdline parameter '-NoSubmit' found, the changelist will be virtualized but not submitted!
LogVirtualizationTool: Display: Attempting to virtualize changelist '28'
LogVirtualizationTool: Display: Attempting to parse changelist '28' in workspace '*******'
LogVirtualizationTool: Display: Found '2623' package file(s)
LogVirtualizationTool: Display: Sorting files by project...
LogVirtualizationTool: Display: The package files are associated with '1' projects(s)
LogVirtualizationTool: Display: Initialization complete!
LogVirtualizationTool: Display: Running the virtualization process...
LogVirtualizationTool: Display: Checking package(s) for the project 'SlayAnimationSample'...
LogPackageName: Display: FPackageName: Mount point added: '../../../../../Perforce/*******/SlayAnimationSample/Content/' mounted to '/SlayAnimationSample/'
LogVirtualization: Display: VirtualizationSystem name found in ini file: Default
LogVirtualization: Display: Initializing the virtualization manager...
LogVirtualization: Display: Loading virtualization manager settings from config files...
LogVirtualization: Display: EnablePayloadVirtualization : true
LogVirtualization: Display: CachePulledPayloads : true
LogVirtualization: Display: MinPayloadLength : 0
LogVirtualization: Display: BackendGraphName : VABackendGraph_Example
LogVirtualization: Display: VirtualizationProcessTag : #virtualized
LogVirtualization: Display: FilterMode : OptOut
LogVirtualization: Display: FilterEngineContent : true
LogVirtualization: Display: FilterEnginePluginContent : true
LogVirtualization: Display: FilterMapContent : true
LogVirtualization: Display: Virtualization is disabled for payloads of the following assets:
LogVirtualization: Display: StaticMesh
LogVirtualization: Display: SoundWave
LogVirtualization: Display: AllowSubmitIfVirtualizationFailed : false
LogVirtualization: Display: LazyInitConnections : false
LogVirtualization: Display: Mounting virtualization backend graph: 'VABackendGraph_Example'
LogVirtualization: Display: The 'CacheStorageHierarchy' has 1 backend(s)
LogVirtualization: Display: Mounting backend entry 'DDCCache'
LogVirtualization: Display: The 'PersistentStorageHierarchy' has 1 backend(s)
LogVirtualization: Display: Mounting backend entry 'SourceControlCache'
LogVirtualization: Display: Virtualization manager initialization completed
LogVirtualization: Display: Considering 2623 file(s) for virtualization
LogVirtualization: Display: Found 2623 package(s), 2 of which had payload trailers
LogVirtualization: Display: Found 0 locally stored payload(s) in 0 package(s) that maybe need to be virtualized
LogVirtualization: Display: Pushed 0 payload(s) to cached storage
LogVirtualization: Display: Pushed 0 payload(s) to EStorageType::Persistent storage
LogVirtualization: Display: 0 package(s) had their trailer container modified and need to be updated
LogVirtualizationTool: Display: Check complete
LogVirtualizationTool: Display: Skipping submit of changelist '28' due to cmdline options
LogVirtualizationTool: Display: UnrealVirtualizationTool ran successfully
Hello, I’m writing here my researchs on the subject. Please, note that I am maybe wrong.
So a payload is the data in package .uassets files. This is the bulk data that the documentation is speaking of. We can find some informations in Engine\Source\Runtime\CoreUObject\Public\UObject\PackageTrailer.h on the subject.
For asset virtualization, the file payload_metainfo.txt regroups all perforce depot synchronized assets concerning with the virtualization. It is filled with command p4 print -q {depot_stream}/payload_metainfo.txt. I suppose it is used by external scripts to know which asset is virtualized.
The UnrealVirtualizationTool is searching for a specific condition. There are different reasons for which an asset can’t be virtualized :
The asset type of the owning UObject ;
The path of the owning UPackage ;
The payload size is below the minimum size for virtualization ;
The owning editor bulkdata had virtualization disabled programmatically ;
The package is either a UMap or the owning editor bulkdata is under a UMapBuildDataRegistry.
These informations are found in Engine\Source\Runtime\Core\Public\Virtualization\VirtualizationTypes.h and Engine\Source\Runtime\CoreUObject\Private\UObject\PackageTrailer.cpp:807.
I don’t understand the details yet. I continue searching on the subject.
I’m surprised that there isn’t any package that can be virtualized in the test project. I am sure that I overlooked something.
The purpose of the ‘payload_metainfo.txt’ file is to give an indication that the depot path is correct. The most common way a package will be virtualized is when a user submits their packages to source control via the editor and it will happen seamlessly behind the scenes, so we didn’t want users submitting their virtualized data to the wrong location accidently without realizing it. When the source control backend is initialized we check for the ‘payload_metainfo.txt’ at the root of depot path (as provided by the config file setup of the project) and attempt to download it via p4 print. This lets us check that:
a) The file exists and so the depot path is probably correct.
b) The user has the correct perforce permissions to access the data there.
As noted above, the expected workflow is to virtualize packages when submitting via the editor. The UnrealVirtualizationTool was originally added as a tool that we hooked into p4v via our custom tool extension, P4VUtils which can be found under the ‘Extras’ directory of the engine. Due to this, the tool can be a little flaky and doesn’t have the best stability/log output and wasn’t documented for the virtualized assets beta release in 5.1. However due to increased interest in using the tool improvements have been made in both 5.2 and 5.3 to try and turn it into a more general purpose way to deal with the virtualization system, but until 5.3 (when hopefully we will document the tool properly) you might want to stick to submitting via the editor.
The most likely reason why you are not seeing and virtualization occurring is that the system requires that the packages be in a relatively recent format, version 1002 or above, which corresponds to the 5.0 release. The reason for this is that we need to move the large payloads from older style bulkdata storage to the newer style format as described in PackageTrailer.h. It is this new storage format that lets us down the virtualization work in a cutdown tool like UnrealVirtualizationTool rather than requiring the full blown editor. If you were submitting via the editor you would not notice the problem as packages being submitted via the editor tend to get re-saved before this happens bringing them to the most recent package version and in more recent versions of the UnrealVirtualizationTool you will be notified in the log output when attempting to virtualize packages in older formats.
So try re-saving the packages in the project first, either with the ResavePackages commandlet or just doing so from within the editor and see if that helps.
Thank you very much @Paul ! That helped me a lot !
Indeed the assets were not at version 1002 or above. The ResavePackages commandlet resolved my problem.
In anycase where the tool creates a non-stream Workspace that may block anywho searching a solution, add the ClientStream in your [VABackendGraph] Config, in the SourceControlCache value, such as
See documentation and FSourceControlBackend::TryApplySettingsFromConfigFiles method implementation in Source\Developer\Virtualization\Private\VirtualizationSourceControlBackend.cpp for more details.