See the above, this is seemingly reporting we have 17GB of unused allocated memory, correct? Which is crazy high to my eyes.
Our situation: Ark Survival Ascended, official dedicated server, after initially loading a very heavy save game.
We did pepper some of the loading code with called to GEngine->TrimMemory() and that helped to reclaim maybe 1GB or so.
But is this just fragmentation? Is there an easy way to tell (I am guessing if TrimMemory doesn’t trim much, it means fragmentation!)
Any ideas to keep it from fragmenting so much? (perhaps during the load we need to call TrimMemory more agressively or… any other ideas?)
Thanks in advance!
p.s. Oh! I also merged in some reporting fixes to FMallocUnused from your Main perforce stream (that was not part of 5.5), so these should be accurate numbers, just FYI.
with a binned allocator I would expect the overhead to be at least significantly smaller compared to a non-binned one.
If this happens with a binned allocator, it could indicate that there are many large allocations that don’t use binning or many small allocations that don’t fit into the existing bins.
Please provide more info on your current platform and allocator, ideally with a comparison between the other allocators.
Do you run your dedicated server on Windows or Linux?
Have you tried comparing -ansimalloc, -binnedmalloc2 (on Windows there’s also -binnedmalloc3) and possibly -mimalloc to see how the different allocators behave?
It might be worth switching to a different default allocator if it fits better for your memory patterns. If none make a significant difference I can get you some advice from our team on debugging the allocation patterns and potentially customizing the bins to better fit your allocations.
glad to hear you’ve found a way to avoid the fragmentation and thanks for the additional info!
One thing to look into might be some kind of pooling for objects/actors/components, as that can help reuse existing objects and reduce the fragmentation incurred if you have a large amounts objects being created/deleted at once.
Since in your case this only happens at loading, you’re probably good to go with your current solution, but I wanted to mention it anyway.
Unfortunately we don’t have any built-in solutions for pooling, but it’s a common thing that licensees implement if they have a heavy object churn.
It’s the binned2 allocator. Yes, I’ve tried with others like binned3, mimalloc, and ansimalloc and none seemed to help much at all (or ended up w/ slightly higher memory usage overall)
I have determined when this happens. It’s during our load of a save game, as the dedi is starting up. We pre-allocate many actors as they load from the save game data before we serialize each actor (this way any actor to actor refs can resolve correctly)
Once we’re done loading, we do a GC and this GC causes us to go from 3GB FMallocUnused up to 18+GB FMallocUnused… which makes sense if we garbage collect so much but it kind of fragments memory. It would be nicer if the dedi’s overall mem footprint could drop by the 18GB just to relieve the memory pressure (as the dedi ends up in mid-high 60GB range otherwise for these popular servers)
I suppose as new allocations happen there is plenty of space in that 18GB FMallocUnused area to allocate from, but it’s still just memory we can’t seem to clear out ever.
One thing I’m going to try is hooking into the GC with GUObjectArray.AddUObjectDeleteListener so as UObjects are deleted I could perhaps collect some info on what exactly is deleting - my hope is we can figure out a way to delete them sooner perhaps to help reduce the fragmenation?
If you have other advice I’m open to all ideas
Oh I did try and tweak the bins a bit, I tried setting AGGRESSIVE_MEMORY_SAVING but that did not seem to help much, if at all.
I think we’ve fixed this. Mostly the issue was due to how we load up the save game (and shows itself when the save game is HUGE)
Basically we do this thing where, when players are not near say our NPCs (Dinos) or building/structure actors we “stasis” them meaning they stay in memory but they don’t tick etc.
Well a recent change to help memory on servers was to delete various components (and recreate them) on stasis. So on loading we end up doing this a lot as most things go into stasis when a server boots.
Fix was to know we’re currently in the middle of a load and just skip creating those objects assuming they won’t be needed.
This helps memory from fragmenting too much, saved us around 12GB of memory.
