Should I really use specular map or leave it at default 0.5 value?

Yes but the problem is not roughness here. You can set your roughness to 1.0, you will still get some specular reflections.
Roughness is clamped to 1.0 (I mean that you can’t go above 1.0 which is 100% rough) internaly so composite texture won’t reduce those reflections, it just helps keeping details in roughness map.