Top is what we can do now, blur a rectangle (on the left is blur result, on the right - blur mask)
Bottom - is what we are trying to achieve - a rectangular area with circle cut angles, and it’s mask on the right.
(but actually our mask looks a little bit different, but I struggled to draw it in paint.net )