So, first of all - disclaimer: This post will be just my opinions, speculations and ideas. I do not have a universal manual on how to do the thing you are trying to do, nor do I know wheather or not there is a “right way” to do it. But I can simply offer my point of view.
Wha I think is, you have to make a game design decision: Do you want your cars to feel like real ones, just scaled down relative to the world so it looks like models (let’s call this Option “A”), or do you want them to feel like toy cars (Option “B”)?
For Option “A”: I would keep the car real-scaled, including all the physics, and I would scale up the rest of the world. Why? I think the Option “B” parahraph will actualy answer this 
Option “B”: I would scale the model down for this option. But scaling the dimensions and weights is far from enough. And I don’t think the chaos vehicle physics is in fault here. To make the car feel like a model, you should replicate more of the model-like properties. Those diecast models, for example, have no suspension, so I would definitely make the wheels just pinned to the chasis. If I’d like to keep some suspension, I imagine there would be a ton of tweeking to make it feel right. I guess the forces in the suspension will not scale linearly, so just changing the weights will not solve it. Also the center of mas will have a different location on your model, torques/velocities/dampings/frictions, all that will have to be tweeked probably, to achieve a “toycar” feeling, if that’s what you are after. I would just try to replicate how a model car is put together and what forces are applied on it when driving, because just scaling a real car down is far from how model cars work…