@presto423
Thank you for the suggestions! I tried out your solutions but sadly it didnt work until I realized you mentioned PNG. I use this because its compressed and doesnt seem to lose much data but for some real it breaks my texture. I exported in TGA and my issue has been fixed! Here are the before and after:
@Arkirsa
Thanks for this suggestion as well, the only reason I didnt have it turned on was because I saw that it didnt desaturate my base colour map.