So I tried exactly what you had specified there, but for some reason my mesh (a big bulky tank) will come within raycast range and then begin flipping between the + and - side of the collision for each raycast. Resulting in the tank turning sideways, and the world being inverted repeatedly until I quit.