reduce reflection / specular in the Skylight

Im using 5.3

In my scene I have a Skylight, and Im replicating the light of a real background I recorded.

To get the same intensity in the shadow areas, my skylight is using a intensity of 9
looks great but the specular in my objects are too intense, and I have shiny objects in areas where should be black.

There is a way to reduce or eliminate the reflection / specular generated by the skylight?

for example in PathTracing looks ok, but I want to try to make it work in Lumen because of the performance