Okay, I’ve figured it out. I did add a second ball as an invisible “hitbox”, and turned off physics simulation altogether and did my own manual physics calculations for speed, accel, gravity, etc. Turning off the physics simulation for my Pawn components allowed me to control them directly while still having collisions.
An important thing to note here that threw me off was that I need to make my terrain/walls etc. have physics simulation turned ON in order for collisions to still work in this set up. Which is… really weird, but it’s fine.
Seems like a horrifying solution but in my particular case I was going to do all of this anyway for my glide racer game, so it’s no pain.