Heyy @Dowslain!
I can definitely see what you are doing here. I think what you may want to use instead in this case (especially if you want to eventually make rooms bigger on the inside) is a bit of smoke and mirrors.
I suggest making dummy rooms, or even have your designated room in a different location, and use render targets and possibly smooth transition portals to achieve your effects. Your dummy room will not have any objects you do not want to show, so that way when your players look at the render it does not show up
There are definitely some Marketplace items that can achieve this, and here are some great non-Epic affiliated resources on how to achieve this:
I hope the above contains the solution you need.