It looks like there is some kind of texturing filtering on the mask that is giving it rounded corners. Is there a way to achieve square holes in the terrain when using the visibility tool, like switching the mask texture to nearest neighbor filtering?