Ok, so I figured this out eventually.
I have 3 animations; aim_up, aim_middle and aim_down. You have to double click each animation and give it an animation to reference:
The problem was, I was making them all reference the aim_middle animation, like this:
What you’re supposed to do is reference the stand animation that you use for your character, like this:
I’ll post step by step notes on how to set up aim offset. All the tutorials and guides out there are unnecessarily complicated.