Ok, so tested some more and seems that the issue is with when the player gets assigned to a team. If you handle the team assignment manually in the “OnPlayerAdded” it should be ok.
I tested with handling the new player within the “OnPlayerSpawn”, just checked if the player who spawns was stored in the “TeamMap”, if not I created a new entry for that player and granted the starter weapon.
Things seems to work fine but I have had some weird stuff still happening when the new player first spawns in and use a different team spawner though so who knows.