Sorry for the late response, one of the only things I can still think of is destroying the old character when switching or disabling it temporary. That may be why teleporting one of the character seems to work.
(If you want switching the characters to keep being possible I would opt for disabling it instead of destroying and creating every time)