For each of your ‘Channels’ you have to treat them exclusively… for instance R is actually R1 G0 B0, R+B is R1 G0 B1
To test for red, R channel subtract G then subtract B then Floor then Saturate. This will give 1 for pure red. 0 for anything else. Use this mask in a multiply or lerp alpha for your color.
To test for R+B, R channel subtract G then add B/ divide by 2, Floor, Saturate. This will give 1 for full Red and Blue but anything less, or if you have any G then 0.
Alpha is it’s own channel. So that’s easy.
Watch for ‘fringing’ along the edges of masks, best to keep this effect along the edges of UV islands. You may need to add a tiny amount like 0.01 before Floor to account for bit depth errors, esp on mobile.
Use ‘Add Named Reroute..’ for your input image RGB channels from your image to make things easier and avoid spaghetti.
Compress your mask image as something like VectorDisplacementMap or Alpha and you may need to turn off sRGB.
If you want even more, there is also the technique of splitting the bits, so you can get 8 masks per channel. You can use this Material Function:
plug your channel into alpha, and the bit value you want to extract. For BitValue use only 0,1,2,4,8,16,32,64,128
You could feasibly get 64 masks out of a single RGB image (turn off sRGB), or just use this technique in the Alpha channel if you want to still use sRGB for the rest of the material.
In my image editor, I create black/white masks for each of these channels, then in Levels, make the max output for that layer the bit value (ie 32). Finally you have to use blending mode Add for all these layers to create the final output channel, use the info tool to check your bit numbers. Takes a bit of workflow but it gets the most squeeze out of a channel.
If using Gimp, you have to use the ‘Legacy’ Add layer blend mode for it to work.
