Teleporting player holding object is not working

I’m trying to teleport the player while he’s holding an object.
I can teleport the player alone, and the object alone, that works fine.
But as soon as I try to teleport both while the player is holding an object, the object moves unexpectedly.
I would assume it’s because of the PhysicsHandle I’m using to grab the object.
I guess I would need some way to temporily disable the PhysicsHandle, move the player and the object and re-attach the object to the PhysicsHandle.
I hope someone can help me solve this issue.

Here are some screenshots:

00 - Setup:
2 Portals. What goes through one portal, comes out the other portal.

01 - At src portal:
I enter the source portal holding the monkey.

My blueprint code moves the player and then the object using SetActorTransform and re-applying the objects velocity using SetPhysicsLinearVelocity. This code works absolutely fine when I throw an object through the portal on its own.

02 - teleported fine:
At first the object is at the correct position

03 - monkey goes crazy:
But then the next frame, the monkey jumpes to a different location.

And then the PhysicsHandle moves it back to the correct position in front of the player.

Any advice on what the PhysicsHandle is doing and how to prevent it?

Edit:
I tried to save all data about which object has been grabbed and before updating the object’s location I released the grabbed component and tried to re-attach it after moving the object. The object ends up in the right place, but it just falls to the ground. The PhysicsHandle doesn’t work. The monkey is still grabbed, but its location doesn’t get updated.

Here’s the relevant code:
Saving data about which object is being grabbed when the player presses the “Grab” key:

		FRotator ObjectRotation = ComponentToGrab->GetOwner()->GetActorRotation();
		FVector GrabbedLocation = ComponentToGrab->GetOwner()->GetActorLocation();
		PhysicsHandle->GrabComponentAtLocationWithRotation(ComponentToGrab, HitResult.BoneName, GrabbedLocation, ObjectRotation);

		MemoGrabbedComponent = ComponentToGrab;
		MemoBoneName = HitResult.BoneName;
		MemoGrabbedLocation = GrabbedLocation;
		MemoObjectRotation = ObjectRotation;

Releasing the component before moving the object:

void UFPGrabber::ReleaseTemporarily() 
{
	if (!PhysicsHandle) { return; }
	PhysicsHandle->ReleaseComponent();
}

And re-attaching the component to the PhysicsHandle after moving the object:

void UFPGrabber::GrabAfterTemporaryRelease() 
{
	PhysicsHandle->GrabComponentAtLocationWithRotation(MemoGrabbedComponent, MemoBoneName, MemoGrabbedLocation, MemoObjectRotation);
}

Updating the grabbed object’s location every tick:

void UFPGrabber::TickComponent(float DeltaTime, ELevelTick TickType, FActorComponentTickFunction* ThisTickFunction)
{
	Super::TickComponent(DeltaTime, TickType, ThisTickFunction);

	//if physics handle exists, update position of the object we are holding
	if (PhysicsHandle != nullptr)
	{
		if (PhysicsHandle->GrabbedComponent != nullptr)
		{
			UE_LOG(LogTemp, Display, TEXT("Updating location"))
			PhysicsHandle->SetTargetLocation(GetReachLineEnd());
		}
	}
}