Actor not destroyed on Widget Button clicked from Client

Hey everyone, I am trying to destroy an actor by clicking a widget button attached to the actor from a client but nothing happens. I am calling Server RPC on click. Any idea how to go around this?

void UBaseInteractable::OnGOButtonClicked()
{
    Server_PickupItem();
}

void UBaseInteractable::Server_PickupItem_Implementation()
{
    Multicast_PickupItem();
}

void UBaseInteractable::Multicast_PickupItem_Implementation()
{
    if (Owner)
    {
        bool bDestroyed = Owner->Destroy();
        GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Red, FString::Printf(TEXT("Picking up: %d"), bDestroyed));
    }
}

Message is printed on screen and owner exists.