It was actually two problems hiding as one.
-
The “dragging effect”, this was caused by the (I believe) pawn controller and something else fighting for control of the camera. To fix this, go to the character settings and check “Use pawn controller for pitch” (“use pawn controller for yaw” should already be checked, that’s why they were behaving differently!)
-
The aspect ratio for my editor view was different than the aspect ratio for the camera I created (the aspect ratio of the two cameras was the same, it was specifically the editor window that was different) which caused a weird shift on the pitch because FOV is normalized by horizontal, not vertical FOV. When I launched the game and it went to its native resolution, the tracking was perfect.