Thanks for giving it a try. I got the solution, it was sort of the sweep, sort of not. What happened was that my forward vector would sometimes aim down a bit. So with sweep it would collide with the ground and stop. But with sweep off it would sometimes glitch through the ground. The solution ended up being quite simple cause the problem was as always quite small and stupid. 
