if instead of baking, you would use layered control rig, then animation fits better. but yea baking is issue