Vertex layers use more instructions on the vertex shader, while texture masks are handled in the pixel shader. Typically, vertices are better to use because there will for the most part be more pixels to render an object than vertices, especially in HD. This is why people say the more you can switch to vertex operations, the better. However, vertex blending doesn’t look good unless you combine it with a pixel texture, so there should be some pixel cost no matter which method you decide to use. Also, vertices are managed all at once, whether the mesh shows up as a single pixel or fills the screen, the cost will be the same. You have to weigh your options to figure out what works well for the look you are trying to achieve.
Of course, just blending one layer isn’t such a huge deal. The more layers you have, the more “tests” the operation has to perform to achieve the final result. Ideally, one would use landscapes to limit the number of layers based on what is painted in the system. You can have 10 different layers set up, but if a landscape component only uses two, then the shader is written only to test between the two layers, not all 10 at the same time. Blending between 3 or 4 materials shouldn’t be too bad, but you will end up with a wide variety of choices doing a linear blend between all of them. See how small you can manage, and if you need more complexity, see if you can find some way to split it up. If not, then see what is the cheapest way you can achieve the result, either simple vertex blending (awful choice, but good on performance), or a basic texture blend (looks much better, but worse on performance).
Normally I wouldn’t stress instruction counts too much because of how incredible modern day GPUs can run, but if gamers demand higher resolutions and performance in 3D with landscape materials that fill the screen, and run it all on a mobile device, you definitely need to watch out!