Can't Figure out how to keep my Controller from spinning

I have a Character Controller I’m making where I start simulating physics on the character and applying force to the collider. I’m having issues controlling the rotation of the character when physics is on due to it rotating uncontrollably.
Here’s a video to demonstrate.

I tried working with Constraints, but any constrains seem to mess up the Left/Right steering. Am I just going about this whole controller wrong? Should I just custom code two separate controllers and swap between them?