Jagged edges on alpha output of Stencil Layer


Hello,

This is how one of my output Layers looks like. Does anyone know what causes this?
here are my settings:

Thank you so much!

Tim