UDestructibleComponent SetRelativeScale3D makes the mesh blink and doesn't update collision volume

I have added a UDestructibleComponent to my class like this(Root is a USceneComponent created in the base class):

ASpellMeteor::ASpellMeteor()
{
    DestructibleComponent = CreateDefaultSubobject<UDestructibleComponent>(TEXT("Destructible Mesh"));
    DestructibleComponent->AttachToComponent(Root, FAttachmentTransformRules::KeepRelativeTransform);
}

Later I set its scale and added a OnHit method

void ASpellMeteor::BeginPlay()
{
    Super::BeginPlay();
	OnActorHit.AddDynamic(this, &ASpellMeteor::OnMeteorHit);
    DestructibleComponent->SetRelativeLocation(FVector(0.0f,0.0f, SpawnHeight));
    DestructibleComponent->SetWorldScale3D(FVector(Size));
}

void ASpellMeteor::OnMeteorHit(AActor* SelfActor, AActor* OtherActor, FVector NormalImpulse, const FHitResult& Hit)
{
    DestructibleComponent->ApplyDamage(1.0f, DestructibleComponent->GetComponentLocation(), NormalImpulse, 100.0f);
    OnActorHit.RemoveDynamic(this, &ASpellMeteor::OnMeteorHit);
}

But when I set the scale in blueprint to something bigger than 2 the mesh starts blinking or just turns invisible. Also when I rush into it i just overlap it without generating hit events untill i reach the middle where the mesh of scale 1.0 would have been.