So, after digging into source of the engine and not really finding anything I decided to go on a random search for functions and I suddenly noticed AddInstanceComponent();
This makes the components stay.
To make this work do the following inside an AActor:
UActorComponent* NewComp = NewObject<UActorComponent>(this, *NewName);
NewComp->RegisterComponent();
this->AddInstanceComponent(NewComp);