Is masking performance costly?

You can enable mask in early depth only in project settings to have zero overdraw on masked objects, provided that you are using early z pass.