Unable to get virtual assets to work in Unreal 5.3

michor.lu · September 18, 2023, 3:27pm

Hey

I have been looking at making use of Virtual assets and I have been following this quick start guide - Virtual Assets Quickstart in Unreal Engine | Unreal Engine 5.3 Documentation

I have gotten it to the point where the editor is attempting to save and check in the virtual asset but just getting an error when trying to check in.

It seems like there is an attempt to save the virtualized payload in the Saved/VASubmission/ folder but when I look in there it is empty.

Anyone got Virtual assets working in 5.3?

I’ve included logs below of what I am getting.

LogSlate: Window 'Submit Files' being destroyed
LogSourceControl: Attempting 'p4 fstat -Or C:/Unreal/Projects/pipeline/Content/pipeline/Test/Grey_Taupe_Gusto_Natural_Stone_bmjbgpieh_8k_Albedo.uasset'
LogSourceControl: Attempting 'p4 diff -sa C:/Unreal/Projects/pipeline/Content/pipeline/Test/Grey_Taupe_Gusto_Natural_Stone_bmjbgpieh_8k_Albedo.uasset'
SourceControl: CommandMessage Command: UpdateStatus, Info: C:/Unreal/Projects/pipeline/Content/pipeline/Test/Grey_Taupe_Gusto_Natural_Stone_bmjbgpieh_8k_Albedo.uasset - file(s) not opened for edit.
LogVirtualization: Display: Considering 1 file(s) for virtualization
LogVirtualization: Display: Found 1 package(s), 1 of which had payload trailers
LogVirtualization: Display: Found 1 locally stored payload(s) in 1 package(s) that maybe need to be virtualized
LogVirtualization: Display: Pushing payload(s) to EStorageType::Cache storage...
LogVirtualization: Verbose: [DDCBackend - DDCCache] Pushed '1' payload(s)
LogVirtualization: Display: Pushed 0 payload(s) to cached storage
LogVirtualization: Display: Pushing payload(s) to EStorageType::Persistent storage...
LogVirtualization: [P4SourceControl - SourceControlCache] Found 1 unique payload(s)
LogSourceControl: Attempting 'p4 fstat -Or //pipeline/mainline/ce/8b/d3/8622a6cd49bd4561baaa1e74839bfbbbb2.upayload'
LogVirtualization: [P4SourceControl - SourceControlCache] Determined that 1 payload(s) require submission to revision control
LogVirtualization: [P4SourceControl - SourceControlCache] Started payload submission session '1500D51743D8BC77DD3BDD86C943F4C2' for '1' payload(s)
LogVirtualization: [P4SourceControl - SourceControlCache] Created directory '../../../../../../Projects/pipeline/Saved/VASubmission/1500d517-43d8-bc77-dd3b-dd86c943f4c2' to submit payloads from
LogSourceControl: Switched workspaces from '' to 'VASubmission-1500d517-43d8-bc77-dd3b-dd86c943f4c2%'
LogVirtualization: [P4SourceControl - SourceControlCache] Splitting the submission into '1' batches
LogVirtualization: [P4SourceControl - SourceControlCache] Processing batch 1/1...
LogVirtualization: Verbose: [P4SourceControl - SourceControlCache] Writing payload to '../../../../../../Projects/pipeline/Saved/VASubmission/1500d517-43d8-bc77-dd3b-dd86c943f4c2/ce/8b/d3/8622a6cd49bd4561baaa1e74839bfbbbb2.upayload' for submission
LogSourceControl: Attempting 'p4 add C:/Unreal/Projects/pipeline/Saved/VASubmission/1500d517-43d8-bc77-dd3b-dd86c943f4c2/ce/8b/d3/8622a6cd49bd4561baaa1e74839bfbbbb2.upayload'
LogSourceControl: Attempting 'p4 reopen -c 3095 C:/Unreal/Projects/pipeline/Saved/VASubmission/1500d517-43d8-bc77-dd3b-dd86c943f4c2/ce/8b/d3/8622a6cd49bd4561baaa1e74839bfbbbb2.upayload'
LogSourceControl: Attempting 'p4 submit -c 3095'
LogSourceControl: Attempting 'p4 reopen -c default C:/Unreal/Projects/pipeline/Saved/VASubmission/1500d517-43d8-bc77-dd3b-dd86c943f4c2/ce/8b/d3/8622a6cd49bd4561baaa1e74839bfbbbb2.upayload'
LogSourceControl: Attempting 'p4 change -d 3095'
SourceControl: Error: CommandMessage Command: CheckIn, Error: No files to submit.
Submit failed -- fix problems above then use 'p4 submit -c 3095'.
LogVirtualization: Error: [P4SourceControl - SourceControlCache] Failed to submit the payload file(s) to revision control
LogSourceControl: Switched workspaces from 'VASubmission-1500d517-43d8-bc77-dd3b-dd86c943f4c2' to '%'
LogSourceControl: Attempting 'p4 client -d VASubmission-1500d517-43d8-bc77-dd3b-dd86c943f4c2'
SourceControl: Error: CommandMessage Command: DeleteWorkspace, Error: Client 'VASubmission-1500d517-43d8-bc77-dd3b-dd86c943f4c2' has files opened. To delete the client, revert any opened files and delete any pending changes first. An administrator may specify -f to force the delete of another user's client.
LogVirtualization: Warning: [P4SourceControl - SourceControlCache] Failed to remove temp workspace 'VASubmission-1500d517-43d8-bc77-dd3b-dd86c943f4c2' please delete manually
LogVirtualization: Error: [P4SourceControl - SourceControlCache] Failed to push '1' payload(s)
LogVirtualization: Error: Failed to push '1' payload(s) to any backend'
LogVirtualization: Display: Virtualization process complete
LogVirtualization: Verbose: Virtualization process took 0.593(s)
SourceControl: Error: Failed to push payloads

Paul · September 19, 2023, 5:41pm

Hello,

This is a problem we haven’t seen before but from your log output either:

The .upayload file is not being written to disk before submission.
The .upayload is being written to the wrong location.
The call to p4 add is failing in an unexpected way that we are not catching.

To help me try to narrow down the problem can you try running the process again then after the error check if the temp workspace mentioned in the log (VASubmission-1500d517-43d8-bc77-dd3b-dd86c943f4c2 in the snippet you posted) actually exists and if so does it have the .upayload in an added state and in that case exactly where is that .upayload on disk?

From your log snippet it does look like the workspace is created and something is in an added state (hence it not being cleaned up) so that should be a good lead.

One final question, is your engine installation in the same disk as your project?

If you feel comfortable you could try running the Microsoft tool Process Monitor (Process Monitor - Sysinternals | Microsoft Learn) to watch for .upayload activity on disk and check if and/or where they are actually being written.

EDIT: You could also try running the editor with -logcmds=“LogVirtualization Verbose” as with verbose logging enabled the virtualization process will log the file paths of each .upayload file written to disk for perforce submission.

michor.lu · September 21, 2023, 1:54pm

Hi Paul,

Thanks for taking the time to respond.

Please see below to some of you questions.

After doing a check-in via the Editor, those temporary workspaces are getting created and appears to be in the add state (see attached). When I use the Perforce client to browse the to file, it takes the projects ./Saved/VASubmission/ folder. It appears to be empty.

The Engine is installed in C:/Unreal/Engine/5.3/Windows/ and the project is located in C:/Unreal/Projects/pipeline/

The log that I have provided do have the Verbose enabled. Do you think there should be more than what I have attached?

Let me get Process Monitor going and get back to you.

Paul · September 22, 2023, 12:50pm

The log that I have provided do have the Verbose enabled. Do you think there should be more than what I have attached?

Sorry that is my mistake, the snippet you provided does indeed contain all the expected verbose logging for 5.3.

Let me get Process Monitor going and get back to you.

This might be your best bet into identifying the problem at the moment. As far as I can tell from your log snippet and replies, everything is working as it should do up until the point the code tries to submit the .upayload file but cannot find it.

At first I wondered if your antivirus system (if you are running one) had removed the .upayload after it was written, but then you should still see the directory structure under \VASubmission\

It could be possible that something is preventing the file from being written in the first place but our file system code should be checking the error values for creating the subdirectories, opening the file handle, writing and closing the handle which would trigger a specific error in the source control backend we are not seeing.

Lastly, in the editor the relative filepaths are usually relative to the editor exe itself, and the paths you provided line up so if the current working directory at the time of writing the .upayload is ‘C:/Unreal/Engine/5.3/Windows/Engine/Binaries/Win64’ then we should be writing the .upayload to ‘C:/Unreal/Projects/pipeline/Saved/VASubmission/1500d517-43d8-bc77-dd3b-dd86c943f4c2/ce/8b/d3/8622a6cd49bd4561baaa1e74839bfbbbb2.upayload’ so I do wonder if the current working directory has been changed unexpectedly at this point.

But all in all, the easiest way to find out the fate of the .upayload is to run Process Monitor and see what it tells you.

moltaaaa · November 15, 2023, 4:39pm

Hey,

I’m wondering if there was ever a solution found for this issue? I am trying to setup virtual assets for UE 5.3 as well and I’ve run into this exact same issue. The process monitor shows that its creating the .upayload files/folders, but those folders are not actually being created on disk. I’m wondering if this is an issue with folder creation within the Editor?

moltaaaa · November 16, 2023, 5:13pm

I wanted to comment in here to update. I solved this issue on my end. It looks like I had the “Saved” folder listed in my P4IGNORE file, so perforce was refusing to add anything in that folder. I removed it from the P4IGNORE file and the virtual assets started working.

Paul · November 21, 2023, 12:21pm

Hello,

So far we’ve identified two different problems that can result in the errors seen in this thread.

a) Some teams wish to store their payloads in a stream based depot, in which case they would need to supply a ‘ClientStream=…’ value when creating the backend in their config file.

b) The P4IGNORE issue that you identified, which can currently be fixed either by adding an exception as you did, using ‘.p4ignore.txt’ as the P4IGNORE value or by informing the backend about the P4IGNORE value by adding ‘IgnoreFile=…’ when creating the backend in the config file.

It seems that our perforce api implementation is not returning proper error strings when files are excluded by the p4ignore file which is making this tricky for teams to debug.

Ideally we will fix this oversight in the api and add the ability for the VA system to be able to poll the P4IGNORE value, so that it can automatically override the file, making ‘IgnoreFile=…’ redundant.

TristonS · December 30, 2024, 1:33am

Using 5.4, I have a similar issue. I have verified P4Ignore is not blocking. I can see the payload asset’s being created inside process viewer. I end up with a librarian error. The closest guess I can make, is that the files are being “added” to perforce, but then deleted before it completes.

I have tried every combination of stream/non-stream, stream1/stream2, stream1:stream1_subdir, and I either get librarian errors, or file not in view errors, both pointing to a perforce command firing AFTER the files are deleted.

They definitely get created, the process otherwise works. It just fails to actually submit. It would be seriously helpful to see a basic setup or something. Downloading some older versions, the class did get updated in 5.4.

		ON_SCOPE_EXIT
		{
			// Clean up the payload file from disk and the temp directories, but we do not need to give errors if any of these operations fail.
			IFileManager::Get().DeleteDirectory(SessionDirectory.ToString(), false, true);

This scope is either firing early, seems more likely

for (const FString& FilePath : FilesToSubmit)
			{
				const bool bRequireExists = false;
				const bool bEvenReadOnly = true;
				const bool bQuiet = false;

				IFileManager::Get().Delete(*FilePath, bRequireExists, bEvenReadOnly, bQuiet);
			}

Okay, so like 5 days later I finally gave up, it works immediately on 5.3. It fails on 5.4, librarian error, the files are added, then deleted, then attempted to be submitted. This is different to the ignore error. Will try get to a bug report.

TLDR: 5.4 STREAMS VIRTUAL ASSETS DO NOT WORK YOU ARE NOT CRAZY

EDIT: Nevermind, I am crazy. It does work in 5.4.

Avoid using a stream depot. If you have a stream depot setup for your development, you can make a second depot in p4d, and just make it a standard non-stream depot. Works immediately without hassle then.

The problem with streams, is it keeps getting conflicted and stuck, whenever a virtualized asset conflicts, so if you update a virtualized file, the payload fails to add with the “file already exists on client” error. This then immediately deletes the temp workspace, so you have no way to easily see what’s going on, and no debug logs make this obvious.

EDIT #2: So, turns out it’s nothing to do with streams, for whatever reason and I am not sure yet if this is specific to my configuration somehow, but I am unable to submit more than 8 files at a time. It will always fail. Made a new post here: Virtual Asset's fail to submit if there is more than 8 files

Hope that helps someone