I use two different approaches depending on the type of structure I’m building.
For a cabin let’s say, since the play area (where the player can move around in) is rather small and a cabin is just a rectangle, I would model the entire thing in a modeling program and import it as one or a few meshes. For the decorations, I would import those separately and place them where they needed to be. In this case, those would be my only modular pieces.
If it was for a sewer level, I would build the play area out of additive and subtractive brushes and use modular decorations to flesh out the details such as pipes, grates, light fixtures etc. If it were an office building, I would approach it the exact same way. The reasons for this is that it can be tested very quickly, you can make changes to the layout easily if needed (you don’t have to reimport an entire hallway just to make a change to it) collisions are automatically in place and as you said, it is infinitely easier to texture.
Do try to avoid the making everything modular method though.