My fix was to disable ‘orient to movement’ in my movement component - which I was previously using for a different movement setup.
When moving backwards it was trying to orient to look backwards, but because im using control rotation for pawn direction, it turns back to the direction im looking. Big Jitter.