It seems to be better now that I lowered reflections resolution for the project. Also, in maps, not using a skylight (here, cheating with a prost-process that adds some color when final image is black at night) frees a lot of memory.
Lumen is sometimes a bit weird. Especially used with Nanite. The memory usage sometimes go crazy and you have to restart UE5.1.1 (version I’m using now but was the same since 5.0) to free it.