Skylight color never goes away?

I am having the same problem using a dynamic day night cycle. The sky light at sunset and noon is far too saturated (orange or blue). Is there anyway to lerp the scene capture with a neutral gray cubemap or something?