In case anyone else runs into this problem, the solution is to use world spaces and parent spaces.
(And to make sure the entire rig is keyed on the frame where that switch happens, or it makes it jittery and finicky!)
This video was also helpful for me since the youtuber explains a lot of what things are whereas others might say “but we’re not dealing with that so we’ll skip the explanation”
