Personally, I do my best to optimize pixel rendering cost. Optimizations here allow games to run faster at higher resolutions and yield the most dramatic performance improvements, especially on middle-higher end hardware. Draw calls are more of a constant overhead that affects lower end users more than anyone else. Still important in D3D11, but no where near as much in D3D12, Vulkan, and Metal. Unfortunately, Vulkan is still not stable enough to develop for right now.
Even if you have a lot of characters you can certainly get away with two material separations, take advantage, and make a nicer, more appropriate cloth shader. But that all depends on your desired fidelity and number of characters: a truly proper character needs a special model for hair, eyes, skin, cloth, and accessories as all these components require radically different shading models from each other. The eye alone typically requires separate shaders too, if you break it down far enough.
How many characters are in your project, what platforms are you targeting, and what fidelity are you trying to achieve?