Hi dokipen -
There is two things you will probably want to do to achieve the look you are going for and maintaining you opacity mask. The first is to make sure that your texture has an unmatted alpha, this is not just an applied alpha channel, but a process which multiplies up the dim values of pixels along your alpha edge to ensure the correct values get translated into the shader. There is a wonderful tutorial that Bill Kladis does which is free to view on youtube, here. You will also want to adjust your clipping value based on the alpha and effect for which you are going.
Masked Materials are always going to be slightly “blocky,” but with the two method above you can achieve something which is much better looking in the long run.
Let me know how it comes out -
Eric Ketchum