Solved it with just setting the location of the camera on the character in PlayerTick with a bool that’s set in camera movement inputs and reversed with a new “focus camera on player” input.
if (CameraMovementRef != nullptr && CameraMovementRef->IsCameraDisabled() && CameraPawnRef && PlayerCharacter)
{
CameraPawnRef->SetActorLocation(PlayerCharacter->GetActorLocation(), false);
}