Hello Sebapler,
I would like to ask you a couple of questions to clarify what you are doing with your scene. You want to dynamically toggle on and off your lights. Are you baking your lights or are you using fully dynamic lighting? Without fully dynamic lighting your lightmass will still be baked onto the meshes in your scene. Be sure your lights are set to dynamic and you have a fully dynamically lit project. This could also be due to any illumination you have associated with your scene captures. If these are static meshes please be sure to set their light map resolution to something like 256. This allows for more accurate light calculations when your mesh is influenced by nearby lighting.
After going through the troubleshoots let me know if a any of these solve your issues.