I just found the solution, which is to disable “live coding” (edit → editor preferences → live coding).
After disabling it, compiling the code, everything stays where it should even after closing and reopening UE.
This is a shame, as it was so comfortable just to press ctrl alt f11 to compile quickly, hopefully this’ll get fixed soon.