Problem is camera is likely attached to the capsule, you really want the camera to be focusing on the Skeletal Mesh. (IE camera component attached to the skeletal mesh).
The way CMC works is, the capsule is always moved, and this is not smooth. The skeletal mesh is what actually is smoothed.
you need to put the camera component on the mesh in the character bp. And toggle it active/inactive (ie spectator camera component, default to inactive), when you are going to spectate, activate it and deactivate any other camera comp. Its a bit tricky, and we did ours via the PlayerCameraManager to avoid such issue.