I solved it by setting bDisableDistanceStreaming=1 , client side on all the levels which are actually streamed in.
Here is sample code for it:
TArray<ULevelStreaming*> levels = me->GetWorld()->GetStreamingLevels();
for (int i = 0; i < levels.Num(); i++)
{
levels[i]->bDisableDistanceStreaming = 1;
}
You will have then to re-enable when the minigame exists.