Direct Specular rendering black with manual exposure

I was doing some testing with the manual exposure and trying to reproduce a photometric result. I’m seeing the direct specular rendering black when I get the luminance of the sun and sky to the proper levels.
The test is using a basic sunshine afternoon day with clouds in the sky. The sky hdr image was created from camera captures and the xrite checker was photographed at the sunlight exposure.
After dialing in the f/11, 1/250 iso100 exposure I cranked up the sun until the xrite white swatch rendered at the same value as the photographed sample. Doing that I came to the conclusion that the directional light intensity is in Candella/m^2. The sky was then dialed in based off of these assumptions and then I checked the with the pixel inspector to verify the intensity. I’m coming out with nit values that look correct and the scene looks great other than the direct specular now rendering black.

Is it possible that the scene is now so bright that it’s out of range? I’m using a directional light intensity of 81,920 which seems really high, but it makes sense math wise.