UE5.2 executable gets stuck at "UGameplayTagsManager::InitializeManager" (just before "Initializing FReadOnlyCVARCache")

Development Programming & Scripting C++
,
1

Hey folks,

I have a memory leak in my UE5.2 code and I can’t debug it. I’m grabbing and saving textures using the code from this repo but after a few thousands images, the executable goes OOM and gets killed by Linux. I’ve already tried Unreal Insights to debug this problem, it says that the untracked memory grows “like a staircase” but nothing more, it can’t pinpoint the source of the problem.

image
image1476×607 42.8 KB

Valgrind is the correct tool for this job, unfortunately I can’t get past the UGameplayTagsManager::InitializeManager line and I never reach Initializing FReadOnlyCVARCache. I already tried using different options and even different Valgrind tools: even nulgrind can’t pass that point.

Attaching gdb I see a lot of looping over ProcessAsyncLoading, but without using valgrind this step completes in a couple of seconds (at worst), I left valgrind run overnight and after 10 hours it hadn’t moved a line.

Anybody knows some nice tricks to discover such kind of memory leaks? What’s the fastest approach in these cases, should I redefine new() and delete()? I’m missing something in Unreal Insights?

Thanks everybody!

2

Ok, I think I solved it… (I should have asked here a week ago, posting problems always triggers mental processes so that eventually all the bugs come home to roost.)

Well the “culprit” was the poor PngImageWrapper that took > 250 ms to compress an image. My loop was 0.1 seconds so images were enqueueing like crazy in the render queue (that was a TQueue, so no way to know its size). Now I changed it to TDeque (that both has .Num() member function and can be push’d/pop’d from both sides).

I still can’t use valgrind, so I had to learn to read better Unreal Insights. Well, it turns out that memory leak queries (the ones with A*B C*) don’t work for me, when I press “Run query” it always says “Tree is empty”, no matter what I do. It took me a while to understand that there wasn’t any “real” memory leak, that LLM Untracked simply means “everything that is not UCLASS, UPROPERTY and so on”. In fact, LLM Textures was rising as well, so IT KNEW where the memory was used!

At the end I noticed that under Tick() a lot of slow stuff was happening (it wasn’t absolutely clear from the beginning, maybe because I wasn’t using -trace=memory when cooking with ./RunUAT.sh? Who knows…). At the end of the day, I finally noticed that FPngImageWrapper::Compress() was painfully slow, no matter the Quality I chose (it’s not even clear if I’m setting it correctly, because it wants a negative number between -1 and -9 and then it flips it to set ZlibLevel, I can’t understand the rationale behind it).

I also had a look under Source/Runtime/ImageWrapper/Private/Formats and I’ve understood that PNG and JPEG are the ones that are “really implemented”. TIFFs, TGAs and so on are just “shells” for their respective formats, there are no compression/decompression routines in there, not even RLE.

So now I’m saving masks as JPEG with Quality == 100, it still produces artifacts so masks won’t be directly usable to train anything, but at least it’s real time and I can inspect everything on the fly. I’ll need a postprocessing step to sanitize the masks (now they have way too many colors inside than the classes in my problem) but I think I already have the python script to do it.