I figured it out. The sampler type was wrong. Normals usually work properly with default settings unless you want to actually do anything with them. Then I have to change compression type to be normal type as well. I knew it was going to be a stupidly obvious answer.
I also flipped the green channel after.
Thanks for the help anyways. It could have fixed a future lighting issue if I were to have one later on.
This person had a similar problem with the sample type: