Warping under glass

After some more testing I’m pretty sure it has to do with the IOR. If I lower it to 0.9 or lower than it fixes the warping issue but then it does not look so much like glass anymore, just looks like a transparent layer