the problem lies in the amount of things you can replicate at once at any given time, and any actor that replicates will keep its network channel open.
I was able to get around the problem by setting my custom-staticmeshactor-spawnable class to have bNetTemporary=true, which causes tear-off in the network. so it’s a spawn and “fire and forget” approach.
and yes I also needed bAlwaysRelevant=true so they would actually get to the clients.
This was enough for me to replicate the entire dungeon generation in my game to clients.
In my case this is enough because I don’t ever need to mess with the actors after they are spawned since they are static structures. In your case if the building parts can be destroyed later or have replicated variables changed this approach might not be enough (since bNetTemporary is const and after tearoff IIRC the link between the server and client version of the actor is lost)