Lighting changes randomly

I’ve encountered this weird issue with the lighting and it’s been driving me crazy trying to fix it. When I first open my project the lighting is super bright, unnaturally so, but as soon as I touch an object in my scene and move it slightly or change anything really the lighting will switch to way darker all of a sudden. Even moving a camera will cause this.

I believe the darker version is how the lights should actually look in my scene. How I set them up. However once I change anything else such as editing a material the lights will suddenly get bright again. I can do this over and over again and my scene keeps getting brighter or darker each time. I don’t get why. I think it may have something to do with ray-tracing which I have enabled. But I never encountered this in other projects with ray-tracing. I even created a whole new project and imported all my assets back in and it still happens.

The thing is I want ray traced shadows. They look more realistic. I just don’t get why this is happening. It feels like a bug but I couldn’t find anyone else experiencing this in the same way. This is making it very difficult to edit materials because every time I change anything in any material or material instance the lighting gets bright again and I have to keep slightly moving something to get it back to normal