All depends on what you are trying to do…
Personally, where I’m limited in both samplers and memory in a game that requires a lot of textures I’d do the following…
I first pack my tangent space normal map into the ‘green’ and ‘alpha’ channels of the normal map as those have the ‘best’ quality in DTX1 compression and least chance of channel crosstalk, then use the red for my roughness channel and the blue for my metallic.
Then when importing this into UE4 you’ll need to disable SRGB and set the compression to DTX1 which i think in ue4 is just called “Default”. Now in your material multiply both the green and alpha channels by 2 and subtract 1, append this with a third value of 1 and plug that into your normal map channel.
There are some things to be aware of though, the first is you will see some artefacts, now if your designing quite a dirty or post apocalyptic looking game you’ll be all good to go. However if you are designing quite a clean game with lots of flat surfaces you’ll definately want to stick with the standard way of doing normal maps and just using an additional texture for masks.
Lastly if you really are strapped for video memory you can pack the normal data into the and green and blue channels while using the red for roughness, you’ll get some minor crosstalk but should still be acceptable in a majority of cases this way you’ll save half the video memory as you don’t need to use the alpha channel.
If you’re just doing this as a minor optimization step though you are best just using 3 RGB maps, the first being base color, second being the default normal map and the third being a 1/2 resolution RGB texture, If you need to use more masks then use the steps above (packing normals into green and blue or alpha channel) and this will free you up either 1 or 2 additional masks.
You should avoid using alpha channels where possible as this doubles the memory usage of the texture and is a bit of a waste except in the above scenarios where you are also limited by the number of samplers but still need to use 2 masks.
edit: Don’t forget about vertex colors, you can use those as masks as well.