While typing this question I finally found the problem in my material. I still posted it so it might help others.
I had a “Transform Vector” node that transformed the noise from world to tangent space. This caused the issue. The node was a leftover from a tutorial.