Texture Visibility Issues in UE when using a shared DDC Cache

Development Programming & Scripting
, ,
1

Hi everyone,

We’re looking for insights with Shared Derived Data Cache (DDC) enabled.

Context

  • Our own plugin: Anima 6.1.0 our plugin streams video frames through a proprietary decoder and presents them as textures on animated raw meshes.
  • Engine: Unreal Engine 5.5.4 (same behaviour on 5.6).
  • DDC Setup:
    • Shared Zen Storage Server on the LAN (default settings, SSD backend).
    • SharedDerivedDataCache.ini points all team machines to that server.
  • Hardware: Two Windows 11 workstations, RTX 40-series GPUs, 64 GB RAM, 10 GbE network.

Symptoms

  • Roughly 60 % of editor launches: animated textures fail to appear—meshes show default grey.
  • Same assets work 100 % of the time if we bypass the shared DDC (local cache only).
  • No warnings in the Output Log beyond the usual “Missing mips at launch, streaming in …”.

User-side Work-around (confirmed on two separate studios)

  1. Disable Zen Storage Server service.
  2. Delete or comment-out the shared-cache block in SharedDerivedDataCache.ini.
  3. Point LocalDerivedDataCache to a folder inside the project root (per-machine).
  4. After this, 4D textures are consistently visible.

What we think is happening

The video frames are packaged as virtual textures during cook; on load, Anima’s runtime fetches them, decodes to GPU memory, and assigns them. Our theory is that some part of the derived-data key for these textures is non-deterministic, so on certain launches the client machine believes the VT page isn’t in the shared cache → texture stays blank until a manual re-save invalidates the DDC entry.

What we’ve tried

  • Re-cooked project with fixed GUIDs for all media assets — issue persists.
  • Set DerivedDataBuildGlobalPostLoad=true to force a rebuild on every open — stabilises but kills load times.
  • Compared Zen logs on good vs. bad boots — no obvious errors; only delta is that failing sessions never request the .ubulk for the 4D textures.

Ask

  1. Has anyone seen derived-data collisions or stale entries with video/VT assets when using a shared Zen cache?
  2. Is there a recommended way to pin or validate VT pages so the editor refuses stale cache data?
  3. Any diagnostic flags we can enable in Zen or the Asset Manager to trace why a particular VT key is skipped?

Any pointers, even just “this sounds like a known bug”, would be greatly appreciated.

Thanks in advance!

2

Hi Victor, this does not match any of our known issues. You could test your theory about non-deterministic cache keys by turning on very verbose logging from the DDC, with the command line argument -LogCmds=“LogDerivedDataCache VeryVerbose” or equivalent. Since you have said that it works when using a local cache with no shared cache, I do not suspect key non-determinism here.

One possibility to consider is whether there are dependencies between cache values or a dependency between the cache value and some other external data. What I mean by that is maybe one cache value contains offsets into another cache value. If the values themselves are non-deterministic, external dependencies like that have the potential to be invalid. You can test for non-determinism of the values by running with -DDC-Verify. That mode can be restricted to specific cache buckets, e.g., -DDC-Verify=Texture, to avoid forcing a rebuild of everything that is accessed. Warnings in that mode will point to possible key non-determinism, and errors will point to possible value non-determinism.

You mentioned that the texture will stay blank until a manual re-save invalidates the DDC entry. Since the DDC is only a cache, its users are expected to regenerate any missing data that they attempt to access. A “get” request that misses is expected to be followed by a “put” request to write the data to the cache. Not following this pattern can lead to fairly drastic performance implications in a hierarchical cache like the DDC.