My team’s project is running into issues where we’re continuously seeing long editor hangs related to perforce calls pulling in virtualized asset data (most notably texture assets). It’s particularly annoying because it seems to happen with each and every mip level of a texture. Here’s the callstack from an insights capture:
Initially this doesn’t surprise me; our DDC setup is non-ideal (we’re in the process of setting up a shared DDC, but right now are only relying on local DDC stores).
I had the team all run locally the DDC fill command in hopes of that’d be the end of it (get past the bulk of our hitches), but after about a month or so the hitches start to creep in again. I’d expect this for new content, but we’re seeing it happen for old untouched textures as well (ones that should have been covered by the fill command).
I am left wondering if something we’re not aware of is invalidating our DDC. As I understand it, there’s no default cache size limit for local caches. What are other ways that we could be unknowingly invalidating our DDC? We haven’t pulled any major engine changes, nor invalidated more macro level things like shader version/hash.
In case it’s relevant, we’re seeing this most notably with asset pack textures we’ve downloaded from the asset store and moved into content-only plugins.
You should not be using virtual assets without a shared cache (or Cloud) . This is the worse case where each machine will have to pull the bulk data for any asset on first use. The shared cache should normally contain the needed DDC data of the assets which prevents pulling the bulk data locally. In this scenario, running the DDC fill command ends up pulling on all the bulk data. All of the data (bulk and DDC) ends up being stored locally in the Zen cache which has a retention policy of 14 days based on the last access of the data. That would explain why the “problem” is coming back for old untouched textures.
In the best case, the workers have a warm shared cache (filestore, Zen (5.6+) or Cloud) that will serve the DDC data. The shared caches are being kept warm by users when they first import\create a new asset and the build farm when the project is cooked.
I forgot to mention that you can change the retention duration of the cache data. It’s the --gc-cache-duration-seconds argument in the ExtraArgs member of the [Zen.AutoLaunch] section in BaseEngine.ini.
Storing the bulk data in the DDC requires setting the CacheStorageHierarchy entry which establish the “connection” to the back end grah.
As far as the “location”, the Zen local cache is now the default (vs the old Local folder). I know the zen local cache is segmented but I’m usure of the details. There are a couple of ways where you can configure where the Zen server stores its data locally (see UE::Zen::DetermineDataPath). We recommend using a single local for all engines and projects so that you can benefit from sharing the DDC data across them.
The local cache needs a retention limit to get rid of the data of older asset revision data. The cache would grow infinitely otherwise. The retention period is based on the last access time of the data so using anything resets its “countdown”.
You can rehydrate uasset using the ‘Rehydrate’ mode of UnrealVirtualizationTool.exe. Running the command without arguments will dump the different modes and their arguments.
Regarding the test with the other project, my guess is that the DDC data was generated during the editor startup and the load of the level. I do believe that the mips are stored separately but I would have to dig the code to be sure. Each type of data has specific code that handles its DDC data.
It is possible to drop (flush) the content of the cache using the command line. In this case, it would be: zen.exe drop --namespace ue.ddc --bucket bulkdata
I guessed that you did not change the default Namespace in the StorageServers section of BaseEngine.ini.
Starting with version 5.6, the Zen Server has a web UI that allows to inspect the content of the cache and or drop data . The default address is: http://localhost:8558/dashboard
That cache config does not use Zen local indeed. You are missing the ZenLocal definition and it should be part of the “Inner” chain. You can check the DerivedDataBackendGraph definition in BaseEngine.ini at the engine level to see how the Zen caches are integrated in the default config.
In this case, the Bulk Data should be stored in the Local folder and will get deleted 12 days after their last access.
UnusedFileAge is the field that decides the retention for the old DDC caches (Local and Shared) .
Bulk Data vs Zen vs DerivedDataBackendGraph: The Bulk Data is stored in the different levels of the DDC as defined through DerivedDataBackendGraph. In your current config, it will be in the Local file cache.
Hi Mike, I’m not quite the expert on this area but I’m going to try to send the right people over to chime in.
It’s normal for the local DDC to cycle through data and things to fall out of cache. Maybe we could figure out some DDC settings that you guys could use to prevent that so things stuck around longer.
It’s also normal for mips to be processed as you fly around in the Editor. Normally this should be very fast. It sounds like it’s not fast for you, and the reason is because the textures source data is virtualized in P4 and that download from P4 is slow. Also any mip that hitches should then go into the DDC and not cause a hitch again.
I did a little digging, and yes the streaming mips are stored as separate DDC objects, so each can be loaded independently, and can be recycled in the DDC cache independently.
If you have plenty of local disk space and the DDC is not making use of it, maybe the easiest fix is to increase the DDC local disk cache size so it’s not trying so hard to recycle things and save space.
“Why does level load cache all mips initially on first level load, but not on subsequent level loads?”
Because if the texture is not found at all in the DDC, then it gets built, which always builds all mip levels and stores them to DDC. OTOH over time if some of the streaming mip levels are dropped from DDC, then at level load time it only requests the texture non-streaming base at level load time, that is found so the texture is not rebuilt, and the streaming mips will not be built until they are encountered.
(this behavior is somewhat modified by the fix CL mentioned above)
Level load shouldn’t be caching all mips as Charles noted, it’s just loading the texture asset and inline mips. Streaming mips are all on demand and get serviced in the editor by the DDC texture mip provider. In there you should be able to see where if it fails to get a mip it’ll call in and force a rebuild of the texture, which would then cause the asset to rehydrate.
I’ll admit I didn’t read the thread too closely, but to be clear, asset hydration should never be needed outside of rebuilding the texture, which should only be needed if the texture gets rebuilt, which more or less shouldn’t ever happen with a properly working DDC.
What my CL above fixed was if you never referenced some texture mips they would get GC’d, and then if you happened to need them when running around in the editor, it would fail to stream them in and do the above mentioned rebuild. In my case this was happening due to LODBias settings preventing the top mips from getting referenced long enough to get GCd in the DDC.
However, my fix only references those mips during a cook, so if you’re not cooking the assets, then that touch will never happen and you would still experience this rebuild.
I would probably start with trying to look at the mip provider code and add some logging to see what mips are missing when you’re flying around the level. That may provide some insight, if my explanation here doesn’t.
FWIW, there’s work scheduled to update the texture ddc usage so that individual mips can’t ever be GCd separately from the texture, however it’s not started yet.
Even if there were I wouldn’t let you believe it, heh. The person who does this work is also one of the main engine firefighters and foundation support crew so he gets pulled in a lot of directions.
Lets try this, can you add this line of code to TexturederivedDataTask.cpp in DDC1_FetchAndFillDerivedData, in the branch that looks like this:
I knew we were working under a non-ideal scenario w.r.t. virtual assets + non-shared DDC. However, I had assumed there was no retention limit to local DDC. I was leaving this on (while our shared DDC gets stood up) to try and narrow down the issues I thought we were encountering.
For clarity, you’re saying both the virtualized bulk data and DDC cache use the same retention policy? Are they stored in the same place? How does the zen cache get mapped to our `DerivedDataBackendGraph` settings (which governs where the local DDC is on disk)?
I recall there being a space limit on the bulk data as well, does that also apply to the local DDC? I see a “low diskspace threshold setting” in the ExtraArgs, is that it?
Also, forgot to ask the best way for us to claw back from virtualized assets. Is there a command to rehydrate everything before turning it off? Seems like the best recourse for us is to disable it until we have a shared DDC (though, we are benefiting from not having to pull down everything, and only the stuff for maps we’re using locally).
We have a second (larger) project at the company that is configured in the same way in terms of DDC/virtualization. They don’t seem to run into the hitches we are encountering.
In an attempt to A/B the two projects, yesterday I deleted my local DDC directory (the one referenced by the `[DerivedDataBackendGraph]` ini setting). After a lengthy editor load, I didn’t encounter any bulk data sync hits while flying around the level. The question I now have is: if everything needed for the map to run smoothly was cached during level/editor load why would I be hitting hitches during level navigation? Shouldn’t subsequent editor loads (after the retention has expired) take the hit during level load?
Following this thought, I guessed that maybe editor level load only requires a particular mip level from the DDC and that maybe other mip levels expire and lose retention? (am I right in assuming that different mip levels have separate DDC records that expire separately?) If this is the case, then I could see that flying around the level would benefit from virtualized bulk data being freshly available in the cache. Is there a way for me to clear just the virtualized bulk data cache? I tried to turn `--gc-cache-duration-seconds` down to 10 seconds, but that didn’t appear to affect anything (maybe the retention duration is applied at the time a cache entry is made?).
As I a dig more into this I believe that we are not actually using the zen cache for the DDC. I tried running the zen.exe command you shared and it complained about missing a server. I was running the editor at the time. This lead me to step through code in relation to how our DDC is configured in our ini.
I never see a zen server spun up in code as a result of parsing our DDC ini settings, which leads back to my earlier question “How does the zen cache get mapped to our `DerivedDataBackendGraph` settings?”.
If our DDC isn’t backed by a zen cache, the bulk data store still may be. Is there a way for me to confirm that? Where would we expect a zenserver to be spun up for that? Would we expect the editor to have an active zen server process running if this were the case? (zen.exe failing to connect would seem to indicate there isn’t an active process to connect to).
Also, curious about my question regarding zen cache and the virtualized bulk data store. Or is that also governed by the `DerivedDataBackendGraph` config?
With respect to textures specifically and how their cache entries are managed, we’re finding it particularly vexing that it seems like certain mip levels of a particular texture are excised from the cache, while others kept. The behavior we’re encountering is:
Run a fill DDC command
Develop in our main level with no problems
Continue to do so for ~12 days
Start hitting texture bulk data pulls when flying around the level (presumably zooming in to textures we didn’t zoom in to in the previous 12 days)
This pattern seems to suggest that some of the cache entries related to the texture are excised while others are not. We can’t expect the user to hit every single mip of every texture in the level in the retention period. IMO, what we want is for all cache entries relating to a single texture (all mip levels) to be retained if any piece of the texture is accessed/used/loaded. Are we maybe missing a setting for this? Maybe one that bundles all mip levels into a single cache entry?