Download

Preloading 70GB of data

Hi.

I’m implementing a cinematic VR experience and need to preload 70gb of data. The catch is that each frame is a model with 200k polys, a 2k texture and an own material (This is the data I received). In order to play the frames like a sequence, I need to preload them. However, I am getting out of memory errors. I’m on a 64GB ram machine btw.

Approaches:

  • Drop all frames in the scene - crash
  • Have one actor, dynamically load the mesh and stream the texture - too slow, crash
  • Put 400 frames of data in sublevels, stream the levels - BEST APPROACH SO FAR, but crashes in the editor with too much frame data

Currently: Trying to use world composition for loading, but it crashes immediately.

I’d be glad for any help.

That sounds like an absolutely horrible inefficient idea. But if you really need to do it this way, why not put each frame in a static mesh asset and async load them a few frames ahead, then swap the mesh each frame?

It’s not like loading levels would do anything different.

Haha. Yeah it is. I got thrown in to help there.

We have exactly done that, but after ~700 frames we still run out of memory.

By now I think the problems are the materials. It might be worth to check how a single dynamic material instance for all models with a swapped out texture parameter for each frame performs. I’m kinda believing at the moment that too many render contexts (= materials, right? correct me if not) are spamming our memory until we run out, even though I would think the GC would take care of that. In case you’re wondering why I didn’t try this earlier - applying a change to enough frames for testing takes like 40 minutes, which is a bit expensive for a good guess.

BTW: The project has been done in Unity before using asset bundles - so sublevel streaming would be the equivalent. It might be worth a try to dynamically add streaming levels, load them, play the frames, unload and remove the streaming level. (Please see OP. We tried this but the persistent level crashes with too many frames in the sublevels, that’s why I’m talking about dynamically adding and removing streaming levels)

Sigh

Are you maybe continually loading new frames, but not unloading the old ones? I believe you can force destroy UObjects like static mesh assets by calling ConditionalBeginDestroy on them, not 100% sure if that’ll work.

It’s worth a try, thank you! I’m back on that on Friday or Monday the latest. I’ll post the solution when we find one. :slight_smile:

You should really do something about that dataset. Minimally, have one character mesh, make sure the point order is consistent frame to frame, store the rest of the frames as point caches. See if you can use one material for the sequence. For the textures if you truly must have a unique texture every frame, see if you can separate the component that is changing (luminance for instance) and store it in a smaller texture. If the points are completely inconsistent frame to frame (indices, uvs, etc) most 3D apps have facilities that can solve this and output a consistent point cache.