Steam Invites when accepted randomly make player restart game

In my game it is set up when a player host a game they can then invite some one through there friends list. The problem is when the friend accepts the invite randomly they will not connect to the lobby and just restart the game. If any one has any ideas I would love to hear them.