Missing PlayerController sometimes after Seamless Travel

Developing a multiplayer game with minigames for 2 years now. Each minigame has its own PlayerController Class. So I swap controller on each Seamless Travel.

Works great 95% of the times. But sometimes I end up with only ex 4 out of 5 players getting their controller. I wait for over a minute and looping through the PC array to check, but nothing.

Is there a way to like reload the process for the players to get their controller or something?
Thank you in advance and I spent so much time trying to fix.

BaseSwap
BaseSwap2