Rather than using the channels as a mask, use the values as a mask. Presumably, you only need a binary value. Something is either brick or not… Concrete or not.
So you decide on standardized values to assign to each material - maybe you paint everything you want to be brick with a value of 1/256 (in any one channel, the other two can be used for whatever else). Then paint everything concrete with a 2/256.
Then, you plug this texture into a bitmask node. You use the same bit value as in input to the bitmask, and it will output a 0 or 1 depending on if the pixel has the bit value specified.
Now you can output up to 256 binary masks from a single texture. Set up the texture array for each building. Each texture will have a specific index in the array, which is specified by the texture coordinates vector input to the sample.
Let’s say you have roof tiles in index 3 of the array.
You also have the bitmask output from the 3/256 value bits in the texture corresponding to roof tiles. Multiply the bitmask by the index value. Now every pixel that will show the tile texture has a value of 3, and anything else has a value of 0. Repeat for as many values as needed, then add them all together.
You will now have a mask that corresponds each pixel’s value with the index of the array that you want that pixel to sample.
Take your texture coordinates, and an append vector node. Append your mask to the 3rd channel of the texture coordinates and sample the array.
This works because of the fact that we don’t need to blend any of the textures together. So there’s no reason to spend more than one bit to say do or don’t apply it there.