I think the issue is due to the wheels not touching the ground anymore while drifting. It happens the same for me when I overjump.
One solution for you is to bring down the center of mass, like -10 or something.
You might also want to play around with the Longitudinal threshold in the Wheels BP.