I figured it out, it has to do with how the MRQ samples each frame. This happens mostly when you have lots of temporal samples. Using many temporal samples, the MRQ samples each frame a bit before and after that frame. For smooth keyframes like moving an actor this is fine, but for binary key frames it causes this issue.
To solve this, in the Movie Render Queue settings, add Camera setting, and set shutter timing to Frame Open.
This honestly seems like a bug, by default binary key frames should always be treated as Frame Open. I’ll submit a bug report about this.