Alpha mask in target layer- causing jagged edges

Hi,

I am using alpha masks connected to my materials in the target layers of my landscape. This makes me control exactly where for example the grass material should be shown.
However, since my landscape tile has a DTM applied to it, which is 1009x1009 pixels, my alpha masks are also 1009x1009 pixels, in order to match. This are causing some wavy jagged edges between the different materials ( see blue arrow), due to the png is rasterized and probably to lowrez.

So my questions are:

Is it possible to use a vector format, instead of png, when importing the alpha masks? If not, would it be possible for someone with programming skills to achieve this?

If stuck with png, is there some workaround to import higher resolution of the masks but still make them to match with the DTM? Maybye some way to rezise them inside Unreal without loosing the details?

Best

Magnus