Unfortunately my Aperture is 6.0 and swapping the focus to track a keyframed Actor instead did not fix the issue for me.
Simply increasing the aperture from 6 to 10 slightly helped. But increasing it further never got rid of the issue. I got to 200 before I gave up. My issue is at least broadly independent from aperture, or how the depth of field is animated.