Something thats not in memory doenst have to be optimized away.
I think using level streaming will help you organize/designing your levels better + you dont have to worry much about memory consumption since the level data is streamed in when needed.