Ok, I think I solved it… (I should have asked here a week ago, posting problems always triggers mental processes so that eventually all the bugs come home to roost.)
Well the “culprit” was the poor PngImageWrapper that took > 250 ms to compress an image. My loop was 0.1 seconds so images were enqueueing like crazy in the render queue (that was a TQueue, so no way to know its size). Now I changed it to TDeque (that both has .Num() member function and can be push’d/pop’d from both sides).
I still can’t use valgrind, so I had to learn to read better Unreal Insights. Well, it turns out that memory leak queries (the ones with A*B C*) don’t work for me, when I press “Run query” it always says “Tree is empty”, no matter what I do. It took me a while to understand that there wasn’t any “real” memory leak, that LLM Untracked simply means “everything that is not UCLASS, UPROPERTY and so on”. In fact, LLM Textures was rising as well, so IT KNEW where the memory was used!
At the end I noticed that under Tick() a lot of slow stuff was happening (it wasn’t absolutely clear from the beginning, maybe because I wasn’t using -trace=memory when cooking with ./RunUAT.sh? Who knows…). At the end of the day, I finally noticed that FPngImageWrapper::Compress() was painfully slow, no matter the Quality I chose (it’s not even clear if I’m setting it correctly, because it wants a negative number between -1 and -9 and then it flips it to set ZlibLevel, I can’t understand the rationale behind it).
I also had a look under Source/Runtime/ImageWrapper/Private/Formats and I’ve understood that PNG and JPEG are the ones that are “really implemented”. TIFFs, TGAs and so on are just “shells” for their respective formats, there are no compression/decompression routines in there, not even RLE.
So now I’m saving masks as JPEG with Quality == 100, it still produces artifacts so masks won’t be directly usable to train anything, but at least it’s real time and I can inspect everything on the fly. I’ll need a postprocessing step to sanitize the masks (now they have way too many colors inside than the classes in my problem) but I think I already have the python script to do it.