when minimized, i have a memory leak in packaged game when only using vulcan or dx12. any ideas?

Development Programming & Scripting
,
1

recently i wanted to update to Vulcan for its better overall performance and compatibility with linux. it was later brought to my attention that minimizing my compiled project when in windowed mode would cause it to have a really bad memory leak, quickly reaching 10gigs+ from only about 500mb in under a minute. additionally, the memory wouldn’t be fully dealt with after, and it would sometimes straight up cause my GPU driver to crash temporarily. does anyone have any ideas why this could be?

Unreal 4.27 source, and tested in a project with essentially nothing in it.

2

you should have a look at:

which when you take the snap shot you want to look for allocations

“most of the time” a memory leak (if that is what is happening) comes from new without a corresponding delete, or like a handle being opened without being closed.

this might require digging into the C++ code base, but if you are compiling from source I would hope someone on your team should be comfortable doing that.

if this is happening in a blank project, and can be traced to one of the engines files I would suggest posting your findings as well as maybe making a bug-report.

3

hmm. im unsure how to look into the C++ because its just me on my project. ill see if i can figure out the issue tho.

as for profiling an already built project, do you have any ideas how i may be able to go about this? the page you sent me to had alot of different pages attached to it, making me unsure which one i should read through.

4

this is a more direct “how to”

and a condensed version:

5

ok. thank you ill take a look

6

Hey have you found a solution for this problem? I’m facing the same issue where when playing on editor project goes normally, but when openning the packaged version and leaving it minimized the memory consumption increases crazily until the game crashes.
Has anyone found the reason and/or a solution?
I’m currently using UE 5.3

7

the “solution” to runaway memory consumption on a given application will typically be unique for each project (this is why I didn’t state any specific “[look here]” type statements)

most of the time finding a memory leak; which in the majority of cases is a dangling pointer (there are others but they are far more rare, and are often caused by errors in the compiler, or problems with Malloc which is Open Source Read). Where the engine runtime is Garbage Collected there could be some situations there, but they are usually explicit.

in full release versions of the Editor/Engine Epic has assured and asked for our “trust” that the Editor’s pre-built blueprint nodes have the memory management taken care of, so you should focus your efforts on looking into your own C++ (the vast majority of dangling pointers will be the use of new) and instead of combing every *.cpp, and *.h in current times we we use Profiling to figure out where these problems are.

even if your project is “only in blueprints” (because your team only ever made blueprint scripts) your plugins, or store assets could have C++, and that could also be the culprit. In this scenario then you will need to report the issue to the plugin/asset creator.

there is the chance that someone at Epic did make a mistake and forgot to de-allocate memory, which especially for a “Blueprint project” will be found through profiling, but that should be submitted as a “bug report” on these forums (these supposedly tie directly into the Epic Engineering Team’s internal tracking, so as “reproducible as possible” would be most appreciated)

8

I too have stumbled upon this problem today in a UE 5.1 shipping build. The application runs all fine, but if you minimize in main menu, GPU memory builds up (seen in Win 11 task manager “Shared GPU memory usage”) until the application crashes with either “Out of video memory” or simply DXGI_ERROR_DEVICE_REMOVED.

I debugged this for a bit now and found the cause in Unreal’s temporal anti-aliasing. It seems that in builds (Debug builds are fine, DebugGame/Testing/Shipping are not), the temporal AA does not behave well when there is not a single mesh on screen. Seems it keeps allocating some buffers in this case - most likely screen buffers, because the memory grows much less quickly if I make the window smaller :slight_smile:

I added a cube to my main menu scene and I could successfully reproduce the issue when I minimized the application while the cube was not visible, and the issue vanished when I minimized the application while the cube was visible.

So, possible solutions:

  • Turn temporal AA off/set anti-aliasing to FXAA by default, only enable it once you’re in a proper 3D scene.
  • Make sure there’s a mesh behind your UI (careful: default player controller allows movement with WASD, so players might move away from the mesh).
  • Find the cause in the engine and fix it :eyes:

@xCANAdan @Hynnax If you’re still battling with this issue, please report back if this solves the issue for you.

1 Like
9

thank you for your insight. unfortunately, that hasnt seemed to solve the issue specifically when using Vulcan, both when running no AA or MSAA, and both when there were no meshes in the scene and meshes in the scene. were you using DX12?
heres the cpp error iv found for reference.
image

going there leads me to this function, specifically the line VERIFYVULKANRESULT(Result);.
overall the function seems to return if the gpu has completed its work or something before more work can be given? im not sure. im interested in setting this to false (or true) to see if i can recreate my memory leak, but doing a single change to my unreal source takes hours to recompile.

	bool FFenceManager::WaitForFence(FFence* Fence, uint64 TimeInNanoseconds)
	{
#if VULKAN_ENABLE_AGGRESSIVE_STATS
		SCOPE_CYCLE_COUNTER(STAT_VulkanWaitFence);
#endif

		check(UsedFences.Contains(Fence));
		check(Fence->State == FFence::EState::NotReady);
		VkResult Result = VulkanRHI::vkWaitForFences(Device->GetInstanceHandle(), 1, &Fence->Handle, true, TimeInNanoseconds);
		switch (Result)
		{
		case VK_SUCCESS:
			Fence->State = FFence::EState::Signaled;
			return true;
		case VK_TIMEOUT:
			break;
		default:
			VERIFYVULKANRESULT(Result);
			break;
		}

		return false;
	}
10

Bummer. We’re indeed on DirectX 12. Things I did to debug this, maybe this helps:

  • Make a debug build, launch it with -llm and then in the in-game console type stat llmfull to see what is allocating memory. In our case, it was listed as “Untracked” but maybe you have more luck. I’m no super-expert on this, but you might want to try both DebugGame and Debug builds if you can and see what info you get. The issue wasn’t present for me in Debug, but it did appear in DebugGame.
  • Try the various showflags in the in-game console, e.g. ShowFlag.AntiAliasing and see if you can pin down what causes the issue. Since resizing the window changed the speed of GPU memory buildup, I was suspecting all sorts of screen-space effects like Depth of Field, SSR/Lumen’s screen traces and eventually Anti-Aliasing/TSR.

Hope this helps. We’re considering switching to Vulkan too, so fingers crossed we figure this out.

// EDIT: I switched our build to Vulkan now and the problem does not happen - I am not sure if my fix solved it, or if it wouldn’t have happened with Vulkan in the first place. Either way, to make sure that you don’t have the same problem that I did, you could set ShowFlag.AntiAliasing 0 and/or r.AntiAliasingQuality 0 from the build’s console and confirm that the issue is still happening.

11

@Daerst thank you very much for your answer.

To be honest none of the possible solutions worked on my project (unless I did them wrong) but changing the project to DirectX11 solved it.
I know its not the best option since I’m losing some features from DX12 but at least I won’t have to deal with my game suddenly crashing when minimized anymore! :grin:

Hope they find what exactly is happening with the engine to solve this.

1 Like
12

ill take a look and see if running a totally barebones install of unreal engine, and a totally barebones project creates the same issue. if it does, then ill post a bug report, and if it doesnt, ill narrow the issue down till i find it

2 Likes