I fixed this. The RG channel of the sampled normal map needs to be rotated with a center of 0 and reversed