Because they use PhysX and only expose a subset of it, you cannot do it on your own unless you can create a fully custom constraint.
The physics engine I’ve had the easiest time building a good car in is ODE. In the ODE physics engine, I do a ray-based tire contact by:
- Specifying a “forward” vector
- Specifying the anisotropic friction flag
- Setting the “forward” and “sideways” friction values
- Setting a “linear motor” for the rolling/braking speed of the tire
- Specifying a little bit of “slip” along the sideways vector, to model tread deformation
(Suspension can also be modeled as a pushing-outwards force with some spring and damping constants)
The ODE library then turns this into the appropriate constraints for the physics solver. IIRC, ODE works in velocity space, which means that the options above can be translated into a desired change-in-velocity for this contact, which in turn goes into the big matrix of all constraints that the engine solves.
The PhysX library does have support for a D6 joint (user-specified rotation and translation allowed) and for a fully custom constraint. If you are using C++, you could start looking there.