This sounds like you definitely are onto something. One of the testers reported that the problem seemed to occure when the client loaded the world faster then the server, and looking at the logs I can see that the playercontroller names/ID are incremented every servertravel.
Im curious how you where able to make a clean project without this issue tho.