I recently found out why my listen server CPU time increases a lot when a client joins (from roughly 5ms on an Ryzen 3900X to 10ms when a client joins). I had around 7000 collectible items on the map, all set to replicate and even though they were set to DormantAll, they still used up around 4.5ms time of the server (GameNetDriver time below in UnrealInsights).
But this was easily solved by not replicating them, since the only reason I set them to Replicate was that when they are collected, they should be removed on all clients. And I thought that the actor needs to replicate so that I can send a reference of it between server and client (client telling the server which item it wants to collect, server telling all the clients which item to remove). But looks like you can still send a reference of a specific actor between server and client, even though that actor does not replicate so it was unnecessary to replicate them.
In progress join also works, since each collectible item has a unique String GUID and I store all collected item GUIDs in a replicated list (I originally needed that replicated list so that clients can save the game as well). So when a client joins, each of their collectible actors checks whether their GUID is in that replicated list, and if so removes itself.