I was able to fix the problem, thx to my m8s at polycount 
My ‘special-map’ texture was (in the Compression Settings) set to Color. Changing it to ‘Mask’ or ‘Grayscale’ (Linear Grayscale in the Sampler Type) fixed my issue instantly.
In addition i got the advice to leave the ‘Specular’ completely unconnected. And to always set my Metallic to exact ‘0’ for non-metals (which i already did by adding my switch).
My final material looks like this:
