I have this problem when cooking my project and I changed these parameters in the engine to 4GB and it keeps giving me an eternal loop where it never finishes cooking.
I managed to solve it add the following in defaulteditor.ini of my project and it worked
[CookSettings]
MemoryMinFreeVirtual=1024
MemoryMinFreePhysical=512
UE 5.4.2 here. 32 Gb RAM. Project’s Content folder is 17 Gb. Infinite loop while cooking.
Solved by closing all apps and running RAMMap64.exe just before cooking to free as much memory as possible (chose Empty in RAMMap).