I’m not sure where you got that idea (I can’t find it in this PBR doc and don’t recall seeing it), and apologies if I misinterpreted what you are saying, but “no diffuse texture should have a median value below 50” does not mean the darkest pixel you can import has a value of 50. Median value is not the minimum value, it’s the point where there are as many pixels in the texture that are brighter as there are pixels that are darker. If you are making your textures brighter in order to raise the darkest pixel above 50, then obviously they will come in brighter than they should. (The same thing will happen if you are just trying to raise the median above 50, but not as badly)
A value of 50 is ~19.5% grey (i.e. dark grey), or 0.195 on a 0 to 1 range. Note that I’m assuming here that you were referring to 50 on the 0-255 scale, rather than a percentage. If you were changing the texture to a minimum value of 50 percent, then your textures would be skewed even brighter, as you are essentially ignoring the entire lower half of the brightness range.
The page linked above lists several non-metallic materials with lower intensity than that: charcoal, asphalt, and bare soil. Green grass is barely above it, and desert sand only a little higher. I also can’t find any reference to median in that page.
The upshot is, if the material you are trying to represent is as dark as, for example, charcoal, then there’s no reason a single pixel in that texture would be brighter than RGB 5, 5, 5 or so (that’s the intensity given for charcoal in the page above, multiplied by 256 in each channel)
Edit: it occurred to me that you might be referring to linear workflow, attempting to offset gamma. If so, perhaps that link will be of use.