Be sure the problem is the collision with the vehicle collider and not wheels.
I mean, i have problems with that too, but with objects that the wheels are not supposed to drive-collide add an exention. In my case i added for small objects like cones other stuff, because if the wheel touches in a bad way, well…
So to be sure is the vehicle collider, make it bigger, extent into the Y axis a lot to be sure if you collide into a barrier the wheels not will touch.
With that and the smooth collision edges , problems gone. Btw, also depends in the barrier line, how much difference between the seams. Because no matter all that if all barriers are in a line at X=1000 position for example, if one is at 1001 or 999 there are chances to collide.
Another thing is the physical material on that barriers. I have set bounciness to 0 and friction to 0.2. You can set all to 0 first to discard all possible. If still is doing that no idea, your best solution will be using a spline, that i whink is the best way or create a custom collider for barriers when track is finished.