Using red and green channels as normal map and using blue as mask?

I need to add a mask channel to my normal map texture. What I would like to do is use the red/green channels for the normal map and have the blue channel for my mask. The question is how do I convert the red/green back to a proper xyz vector for the normal input?

That works, but I was going to use the blue channel for the mask which means the blue magnitude would have to be derived from the red/green magnitudes. However, I wan’t comfortable with the overhead that would entail in the math. So I just went ahead and added an alpha channel and left the RGB channels alone.

Calculating blue channel from red and green is so cheap nowdays, that you would not want even considering this cost. Adding alpha channel however brings you to a point, where using two textures, one with normal map compression, is actually more favorable.

This particular model doesn’t really need a defuse texture which is where I would normally put an alpha mask. So the choice is whether to have two textures (both 2048x2048), or just one texture that holds both the normal map and alpha mask.

Isn’t the blue channel just 0.5, or does it get changed a bit after compression? I would think that you would be able to append 0.5 to the red and green channels to create your normal map.

If you mean that you want to replace the blue channel (Z axis) of a tangent-space normalmap with a constant scalar value, then use 1.0 instead of 0.5. Note that this only applies to tangent-space normalmaps, doesn’t work with world-space normalmaps.

All depends on what you are trying to do…

Personally, where I’m limited in both samplers and memory in a game that requires a lot of textures I’d do the following…

I first pack my tangent space normal map into the ‘green’ and ‘alpha’ channels of the normal map as those have the ‘best’ quality in DTX1 compression and least chance of channel crosstalk, then use the red for my roughness channel and the blue for my metallic.

Then when importing this into UE4 you’ll need to disable SRGB and set the compression to DTX1 which i think in ue4 is just called “Default”. Now in your material multiply both the green and alpha channels by 2 and subtract 1, append this with a third value of 1 and plug that into your normal map channel.

There are some things to be aware of though, the first is you will see some artefacts, now if your designing quite a dirty or post apocalyptic looking game you’ll be all good to go. However if you are designing quite a clean game with lots of flat surfaces you’ll definately want to stick with the standard way of doing normal maps and just using an additional texture for masks.

Lastly if you really are strapped for video memory you can pack the normal data into the and green and blue channels while using the red for roughness, you’ll get some minor crosstalk but should still be acceptable in a majority of cases this way you’ll save half the video memory as you don’t need to use the alpha channel.

If you’re just doing this as a minor optimization step though you are best just using 3 RGB maps, the first being base color, second being the default normal map and the third being a 1/2 resolution RGB texture, If you need to use more masks then use the steps above (packing normals into green and blue or alpha channel) and this will free you up either 1 or 2 additional masks.

You should avoid using alpha channels where possible as this doubles the memory usage of the texture and is a bit of a waste except in the above scenarios where you are also limited by the number of samplers but still need to use 2 masks.

edit: Don’t forget about vertex colors, you can use those as masks as well.

One thing about just setting the blue channel to one is that when the engine normalizes the normal vector, it’s going to mess with the red/green magnitudes. The result is that it will likely wash out the normal map to some extent (especially on highly curved surfaces). The fix is to adjust the blue magnitude so that the normal vector remains unit length. However, this involves taking a square root which may or may not be expensive. I’m not sure so I avoided doing it.

One other consideration I had was that I needed the full 2048x2048 texture on both the normal map and alpha mask. I didn’t want to use two separate textures due to their size, especially since the second texture would only be using 1 channel.

Some problems in the post. DXT1 does not have alpha channel.(there is 1bit alpha channel mode but it’s not supported by engine.) When using rgba texture with default compression it’s using DXT5.
Your normal map encoding is dead wrong. Result is not unit length. Correct way is to use DeriveNormalZ node.(but there is actually bug in the 4.18 so I just use this custom node.

MaterialFloat NormalZ = sqrt(saturate(1.0f - dot( NormalXY, NormalXY) ) );
    return MaterialFloat3(NormalXY.xy, NormalZ);

This is what engine does underhood when using NormalMap compression combined with BC5 compression.(just two channel version of BC4 which is same as alpha channel of dxt5).

Problem with combining other channels with normal map is that you get different quality for x and y. Your method is giving almost perfect quality for y channel but x channel is garbage. Where y channel do get 8-bit endpoints and 8 interpolated values between those. X is getting only 6-bit endpoints and 4interpolated values between those. Also x is getting cross channel talk from roughness and metallic.
More information here. Understanding BCn Texture Compression Formats – Nathan Reed’s coding blog

This is actually really good advice. When using separate textures for separate needs you can reason about texture size a lot more. When there is less artifacts from compression you also can use a lot smaller textures.
Usually I can use 1/4 texture size for metallic map(BC4/Alpha) and emissive.(DXT1). Ambient occlusion is 1/2 or 1/4.
So when you can trust that you get good quality compression you get bigger savings from choosing texture sizes per usage.

Isn’t the “DeriveNormalZ”-Node doing what you want to achieve? So, obtaining the blue channel from red and green (given that both are in -1 to 1 scale).

Yes but there is nasty bug. It’s does not use saturate to clamp sqrt values to be positive. This can cause infinity to pop out which translates to super bright spots with forward rendering or mobile. Then bloom white outs whole screen. I have submitted fix for this.

Ah, that’s interesting! I used that node quite a lot and never got errors like that. But good to know, thanks for pointing it out!