Minigolf physics acting weirdly

If you are applying an impulse to move it make sure you are not applying it to the bottom of the ball - if you are you would be adding back spin.