I’d recommend not destroying the pawn when streaming the levels, just teleport it to the location.
If you are using Streaming Volumes to stream the level in early, then you should be fine.
If you are using using BPs to stream in the level, make sure you set Should Block on Load to true. Only once that completes should you set the location of the pawn, or spawn the new pawn. Make sure to set the player linear velocity to 0 also.