Texture masking is utilizing a certain channel of an image to ‘mask’ out parts of the image so that you can operate only on those parts and leave the rest of it alone. This allows you to perform more complex operations on the texture (via math essentially) rather than texture memory.
Most commonly this is done with the alpha channel however this can also be done with specific RGB color values.
This video should answer your question in detail :
