Host crashing from late joins after a level has been streamed, unloaded then restreamed

So the issue stemmed from me having an actor that simulated physics in the map. After switching it to be dynamically spawned through my game state rather than it being placed in the level manually the crashes stopped.