RTX Skylight shadows don't act as expected

Hey everyone!

I am relatively new to UE4, and pretty new to RTX, and have a question.

I have an external scene that I am using full raytracing on - direct light shadows, AO, reflections and so on.

I am using real world LUX values in my lights, and measuring bounced LUX values off white cards in the scene to make sure my sun and sky values are roughly correct, and have a 5:1 contrast ratio.

I am seeing something odd however when I enable “cast raytraced shadows” on my skylight, my shadow areas get super dark, and it looks like the specular values on materials get boosted in the shadows.

If I crank up the skylight intensity to boost the light in the shadows, then everything ends up looking super reflective.

To add more light into the shadows, I tried enabling brute force RTXGI, but that really honestly didn’t do much on my outside scene.

Anyone got any ideas on how to remove the overly specular material response when using cast raytraced shadows? My materials all have default spec values of 0.5, and have the correct metallic and roughness for the materials.