VInterpTo doesn't smoothly move camera

Hrm. Try sticking this in there around the SetRelativeLocation.

GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Red, finalCamPos.ToString());

and seeing what pops out.