Hey dekkuran -
It is hard to tell from your image but most likely these jaggies are caused by your masked material and a rough alpha mask. I would highly suggest looking into this tutorial by VFX Artist Bill Kladis on creating Alpha Maps for VFX.
Thank You
Eric Ketchum