Along with @'s suggestion, this is what I do:
Have the main map where you’re building your level out.
Have another map you use to build out your brushes.
On the building map, build out your brush pieces and stuff, then MAKE A COPY (important - you lose the brush if you don’t make a copy before this next part) then convert it to a StaticMesh (there’s a button in the Details pane with a brush selected).
Your performance will still degrade if you leave the brushes in your building map, but you can then just disable the live updating of brushes, or save those brushes out and delete them so you’re only working on one at a time.
Go back into your main map, place your static mesh and you’re good.
That way you keep all your brushes out of your main map and prevent them from messing up performance in the editor, but still have the brushes around in order to change them later and still be able to make a new StaticMesh.