For best quality vs memory tradeoffs you really need to understand how these texture compressions work. So start by reading this.
Another thing that affects textures is how you use them. Good example is height map. If you are using for POM you really want it to be in separate texture to save bandwith. POM takes multiple sample from texture so you don’t want to pollute texture caches and waste bandwith for channels that POM isn’t using. So height should be packed to single channel compressed non srgb texture.(Alpha/BC4)
Nice simple trick to know is that if your texture don’t have alpha channel but you read it it’s always 1. So this can be used to save memory without any additional cost for materials runtime or setup cost.
It’s also very important to know that default DXT1(default for rgb) compression can’t be used for data that is not dependant. Also r,g and b does not use same amounts of bits.(5,6,5). AO, roughness and metallic does not depend on each other so you should’t put them to same texture color channels. RGB and A is separately compressed on PC but not on mobile so this also can affect how you choose things. Texture size is also big deal. Metallic and AO can be usually use lot smaller texture size than color or roughness. So those should’t be on same texture because this make them size to be locked to each other.
What we use is this:
For albedo + roughness: Default rgba.(1byte per texel). For fully rough objects check compress without alpha to save 50%.(½byte per texel)
Normal map: Normalmap compression.(1byte per texel)
Metallic: alpha/BC4.(½byte per texel). Size is quarter or even smaller. So actual extra is about 1/8 byte per albedo map texel.
Emissive: Default rgb.(½byte per texel). Size is quarter or even smaller. So actual extra is about 1/8 byte per albedo map texel.
Opacity alpha/BC4.(½byte per texel). Size is quarter. So actual extra is about 1/8 byte per albedo map texel.
Height: alpha/BC4.(½byte per texel). Size is quarter or even smaller. So actual extra is about 1/8 byte per albedo map texel.
This way you get best control for quality vs memory and minimum amount of compression artefacts.