How come some materials darken after light building?

Hello to everyone, I have a particular problem.
After I built the light in my scene I noticed some issue with some materials that were supposed to be (for example) white but they are in a dark grey tone.
Later on, I built the light again from 10 to 80 indirect light bounces and it solved the problem.
But then I had to rebuld the scene another time and I did with the same parameters and the problem came back again.
All normals and uvs are ok. I have a feeling that it has something to do with the HDRI Backdrop and its Skylight.
Thanking in advance for anyone who answer!