Brightness increases when looking down, darkens when looking at light objects?

I have this weird thing where the brightness increases if I look down, it super decreases when there are light colored objects (like the base primitive shapes of white,) and it looks fine in most other cases. I found a setting called EV under the View Mode drop down, but while that affects the strange happenings of brightness changing, it also lowers/increases the overall lighting of the whole area. I want to get rid of this automatic brightness change without having to mess up the viewport lighting itself.

Pictures of the issue attached, showing how drastically the lighting changes when moving around the world.