I have the same question: All the shadows cast by geometry in my scene are grey, but the shadows from the normal map on the materials are black. The highlights, by contrast, are what I expect to see: light hitting the material adds a little highlight. But anywhere where there’s shadows goes straight to black, creating a lot of weird contrast.
The material in this pic is in direct light: See how the highlights at the top of the bricks is nice and subtle, while the shadows at the bottom are black? How do I adjust that?