Third Person Char. Randomly Gets Stuck and Drifts

Okay so I figured something out. The player gets more messed up the farther he is from the origin point of the world. At 0,0 he works fine. I mitigated this by moving the center of the island to the origin. But at the farthest west and east end of the island, the movement is still messed up. I know I have a really large world (a life sized recreation of the island of Crete because I just think it’s neat) but does anyone know why this happens and how I can fix it? I don’t want to scale the world down because it would make the physics and scale inaccurate.