Hm if you’re using a directional light, then that means it is forcing a shadow cast on every object in the scene within a certain distance to the camera. I would recommend turning off “Cast Shadows” on any objects that you know won’t be affected by shadowing (small objects, objects against walls, large exterior architecture that is not casting shadows into the interior, etc). I agree that Lumen is probably your best bet here. You could technically use baked lighting with a moveable directional light, but you would have to deal with seams in the procedural pieces. Better off just using Lumen.
Another trick you can use is to change the “Skylight Leaking” in your postprocess volume. This allows the Lumen GI to cheat additional ambient light into your interiors and can help minimize the number of moveable lights you need to add indoors.
Overall though, sounds like you’re already optimizing your lighting pretty well.