Just a snippet of insight. Meshes are read from drive and stored in memory on demand. If the mesh has LoDs they are loaded in ascending order. Low to high. As well as their materials. If each LoD has a different material (typical), then those are loaded in the same order.
e.g. Mesh 1 LoD 4 mesh, then material, LoD 3 mesh, then material etc until the last lod, then collision.
When you “Prefab” a scene all of it is needed when any part is called. Overall your telling your system to choke down all the pieces of a complex scene in one go. Whether this hitches or not depends on the hardware and overall performance of the platform doing the work. e.g. My PC vs Yours.
I’d profile the prefab, then take steps based on the results. Maybe materials are too complex.
Depending on usage I might attempt to use a trigger to spawn the interior deco. Slap one huge, or multiple trigger boxes/collision boxes around the main building and spawn deco/interactives via player overlap.
Just my 2 cents.