Are you using some sort of dummy actors placed in the level which spawn the actual pawns? Sounds that way since you say they keep spawning over and over any time you get close.
If this is the case, try storing a reference to the spawned pawns as a var on the spawner, then in its event EndPlay, destroy the referenced actors. Dunno if you’re using World Partition in UE5 or World Composition in UE 4, but either way, the EndPlay event on editor placed actors should be triggered when their level is streamed out.