Hi UDN, we need some support with a memory leak problem.
When we leave the application idle, we see that the memory used begins to increase. Using Unreal Insights, we can easily see this increase in the Total Allocated Memory graph.
The following image shows the two points in time, where there is a difference of almost 4 GB
[Image Removed]
But when we investigate these two points, we can’t see any increase or difference in allocated memory.
The following images show the investigation of both points and the memory used, which does not reflect the 4 GB difference.
[Image Removed]
[Image Removed]
Memory Leak Investigation:
[Image Removed]
After reviewing the graphs, none show an increase in memory usage over time.
(We also validate all disabled graphics)
[Image Removed]
[Image Removed]
What can we do to find the memory leak? Why isn’t the almost 4 GB difference recorded in any graph?
The Total Allocated Memory (Min\Max) stats are for the frame and represent the lower\higher usage of RAM that gets sampled a couple of times per frame. The “LLM Total” stat is the amount of RAM used by the process as reported by the OS at the end of the frame. The LLM Total should always be between the TAM Min\Max so that would be a bug.
Can you share the trace so that we can use it to debug Insights? Can you also compare the “Commit Size” of the process from the Details panel of the Task Manager? This should match LLM Total. You will have to add the column by right-clicking on the headers of the columns already there.
We had a look at the trace and found that there are mismatching allocation events. This could explain why the Total Allocated Memory (Min\Max) categories seem to be leaking. I have open a bug report for this.
The LLM Total stat is fairly stable. It is pretty much the amount of RAM used by the process from the OS standpoint. Can you check the “Commit Size” of the process from the Details panel of the Task Manager? This should be close to LLM Total. You will have to add the column by right-clicking on the headers of the columns already there. The commit size would be the ultimate measure to know if the process is leaking.
However, we have a memory leak, which we can see by checking memory usage with Task Manager. This problem becomes evident after several hours of the application in Idle mode.
Do you have any advice on how to find the problem? We can’t use Unreal Insight for more than 20 minutes because it consumes a lot of memory, so using it for 4 or 5 hours is impractical.
The best way to track leaks is usually to create a test scenario where the project cycles between a neutral level and the simulation\game level. For a game, it will involves going back and forth between the main menu and the game. You might need to create an empty menu that simply reloads the main level in your app to reproduce this cycle. The first few iterations will make sure that all buffers, pools and internal structures of the engine have stabilized so they don’t look like leaks. Going back to the main menu does ensure that all the data loaded by the level will get unloaded\destroyed. You can then compare the RAM used by the main menu and see if the game level leaks.
When using insights, it is useful to emit bookmark or regions to be able to know what level is current. They can be emitted from BP:
Trace Bookmark(Name): emits a point-in-time bookmark with the given name
Trace Mark Region Start(Name): begins a named region
Trace Mark Region End(Name): ends a named region
You can then use the Insights Memory Leaks report where A & B are aligned with the game level and C is aligned with the menu on the next cycle or even +2.
Another alternative is to activate the malloc leak reporter. This is an allocator that injects itself in the allocator stack and identify call stacks that are never freed or consistently grow. This requires defining MALLOC_LEAKDETECTION as 1 so it is compiled in the engine. I prefer to modify MallocLeakDetection.h as it doesn’t invalidate the entire codebase. I also change the DefaultLeakReportOptions in the constructor of FMallocLeakReporter so that RateFilter is set to 0. The rate filter will not care for slow leak otherwise.
mallocleak.start: begin capturing the callstacks
mallocleak.stop: stop capturing the callstacks
mallocleak.report: Generate a report under Saved\Profiling. You don’t have to stop the capture to generate a report
In this case, you would do 2-3 menu\level cycles to ensure the engine structures are stable. You would then start capturing in the menu, load the level, run the simulation a bit, go back to the menu and generate a report. The report should contain candidates for leaks. You might need to capture a few cycles if the leak is slow. The default option SizeFilter is set at 512kb.
Feel free to share the traces or reports if you want us to analyze them.
Thanks for the reply. We already have a test scenario and are using the investigation module. But the problem is the time required to reproduce the memory leak, which is more than two hours.
Additionally, we cannot modify the engine; we only work with the vanilla version.
We know that our memory leak happen in the idle mode and it can be appreciated after several hour of running.
I tried today with the console command Trace.SnapshotFile, generating a trace every 60 min. Because it is not possible to run for several hours the normal trace (Need to much memory)
[Image Removed]The image shows the comparison of two traces with a time difference of 2 hours. After that time, i can see a difference in the total memory and also in the LLM Textures.
How can i get more information about the allocated packages (Textures)?, because if i use the investigation modul, there is not much information. Functions and Packages are empty or Unknown.
[Image Removed]Any advice on how we can compare the memory allocated at different times over approximately 4 or 5 hours?
We don’t have tools that compare memory snapshots. You can try memory reports from the console. They are a collection reports that cover multiple categories of assets (StaticMesh, Texture, rendering data…).
Regarding the Function column, it appears that the debugging symbols of the executable are not found. You can try to load them from the Module tab. For the Package column, most allocations are not attributes to packages so it can be difficult to find the ones that are. The best way to view the allocations per package is to add one of the Package grouping modes.
[Image Removed]
The “Unique Values - Package” grouping mode will show allocations per package.
There is one point I forgot to cover. You mention that you suspect the leak coming from textures. If you are using regular texture (not virtual ones), it could just be mips that stay resident in the pool. The texture pool is an LRU cache that will only evict mips when the maximum is reached. It could also be render targets that are kept alive. The RTs have a separate pool. You will get informations on both pool in memreports.