Since refraction bends light rays, actually yes you can see somewhere different when looking through glass compared to if it wasn’t present - just like how you can see around a corner with a mirror, due to the changing direction of light rays as they bounce.
But the reason this is happening in a way that looks unnatural is because you’re probably using screen space refraction (as opposed to ray traced refraction). Screen space refraction is an approximation, and generally shouldn’t be used on large surfaces due to it’s limitations.