But I don't want objects in front of the refractor refracted!

The lower distortion dip is actually in front of the object and should NOT be distorted. How do I keep this distortion from happening?

I have noticed that this is a problem in a LOT of game engines.