So for clarity, everything works with replicating the actors to the client. I set the bNetTemporary flag so it should only replicate when spawned and then close the network channel for that object. Right now I am generating 20-40 rooms to build the level.
Still, it is unnecessary as I can use the seed to deterministically load the rooms on both server and clients. So any ideas would be appreciated.