Here is totally empty game mode. It has a variable, soft object reference to some actor.
Then the size map of the game mode:
(purple is the soft object referenced actor)
Am I misunderstanding, but isnt the point of using soft reference to avoid this?