Here is a hot fix, add a trigger box the the location that far away location, so when the player get teleported to it, it will then teleport the player to the correct location.
If that location is in an area the player can walk in,then make so it ios only active then the player teleports and only for a small amount of time like 1 or 2 seconds.