Sequencer causing performance issues?

I’m creating a realtime experience that has a master level with different sublevels for different scenes, as a well as a master Sequence with different subsequences for each different scene.

However, I’m noticing a major performance drop just from simply having Sequencer open. Even if I hide all my levels, and have just a black empty blank level, my frame rate will be 120 fps until I just open the master sequence, and it immediately drops to 50-70fps, even without hitting play. When I actually unhide levels the becomes of course worse.

What could explain this? Is it because I have too many tracks or subsequences? Do I need to break this up in a different way? Are there any optimization tricks that can help bring back my fps back up to what it should be? I’m unsure why just having the Sequencer open, even without playing, would cause such a drastic hit in performance.