Memory bloat tied to Verse with sudden memory cliffs when loading new areas even when Verse is not running - Streaming enabled

Summary

Quick Summary
A complex experience with 5 big areas with Streaming enabled.
Observing sudden drops in memory (of 70MB) when going into new streamed areas, happening when all Verse Devices are off, so no verse should be running, or when all level design assets are removed.

We are using the spatial profiler specifically on Switch to understand the memory use in our project.
We are able to see the problems as they happen in the “Available Memory” metric. However, the results of our testing put us between a rock and a hard place and have created more questions than they answered.

Our baseline, in its current project state is 450mb available memory on Switch when loading in. (a completely “empty” template project is 1.1gb).

Flying around to different areas and spatially loading them can very quickly put us in the red zone (<200 mb) and after that the crash is a matter of time.
To try to see how much is due to our verse code, we disabled all verse (all ui and bunch of initial setup gone), that recovered us to 550mb.
Deleting ALL non gameplay level design assets (landscapes, buildings etc), and we got lots of them with 5 major levels, puts us at 650mb (only 100 MB of memory).

!! So basically if we run no verse and delete all our level assets we are still trivially able to cause OOM errors. !!

We went on to testing a few more things, and so far the only thing that allowed us to somewhat consistently avoid OOM errors is actually DELETING all verse code (or at the very least deleting all verse devices and untagging all devices). This is consistent with the weird behavior we are getting with the “temperature thermometer” where just mentioning something in verse and not using it is eating memory.

BUT it is not actually the static one-time footprint that is concerning. What we are seeing is having the verse referenced in any way is causing HUGE drops in available memory when flying around. Specifically our infamous 70mb “cliff”. The only way to get rid of said cliff and regain somewhat stable spatial loading behavior is to literally have no verse whatsoever. The 70 mb “cliff” is a behavior we can consistently reproduce by entering certain areas and we spent 4 days trying to isolate the potential culprit to no avail. Normal spatial loading behavior gradually allocates and releases, the “cliff” eats a huge chunk, always same size with very little rhyme or reason and doesn’t recover quickly and is thus one of the prime causes of OOM
We ran tests on both Switch and PS4 (low memory/shared texture memory and ram) and the “cliffs” occurred on both

At this point we are in a bad spot where asset optimization gives us very little and to continue to figure this out one line of code at a time is just not feasible (we already wasted close to a week just running these tests)
We are suspecting that the verse runtime itself, on top of eating base memory is causing some weird behavior together with our setup. We could not find any helpful documentation on this.

Our project is massive and quite complex, and very likely we are working at the frontier of what is capable in UEFN.

Please select what you are reporting on:

Unreal Editor for Fortnite

What Type of Bug are you experiencing?

Memory

Steps to Reproduce

  • Load spatial profiler connected to Switch user or any other low memory device
  • Fly from spawn zone to other zones

Expected Result

  • Since there is no Verse running - no huge memory cliffs
  • lower memory due to new assets loading in and recovering while the older streamed assets unload

Observed Result

  • sudden memory drops of roughly 70mb when Verse is not supposed to run. This is happening even if all visual level assets are being removed

Platform(s)

any platform but mainly noticeable on low memory devices like Switch

Island Code

5652-8758-6723

Upvoted

The status of FORT-1042624 changed to ‘Needs Triage’. We’ve confirmed the issue and it’s waiting to be assigned to someone to fix it.