I’ve done something similar and have had it working well - the only things I had to do was ensure the source map had no MipMaps, and it had no smoothing interpolation (nearest pixel) on the mask.
Try setting both textures compression to VectorDisplacement (has no compression) and interpolation to Nearest Pixel