Should I really use specular map or leave it at default 0.5 value?

Ok so am I wrong when I say that a 100% rough material should have 0% reflectance? I don’t get the relation between the two values and to me, they fight each other.
Like the example I posted earlier, how can a 100% rough material with default specular value can look like this?


It looks like a plastic/dusty surface