So I have the answer to this issue : WAIT
I only needed to add a delay of one second before switching to spectator, because there can be an issue of you try to switch before having a pawn.
Now my authority has a pawn but cannot move (but the clients can), this is another problem though.