So in the end what the problem was is that I was using Substance designer, for anyone in the future who gets this problem, if you find your Substance texture is being funny and you get this issues, just make sure that your Albido/Base Color output bit rate is 8bit and not 16it.
The problem with my texture here was that it was 16bit so I just needed to convert it to 8bit on which you can do in Substance Designer.