I dont think that approach will support large and complex models in terms of performance.
The main bottlenecks I see is: One drawcall, per brick!. Even if you only use meshes at simplest LOD level, a model would still require houndrets of drawcalls.
You could however make several version of the model (as you did with the bricks) and use them also distance based, as blueprints, as you suggested.
Another optimization step here would be not using several material IDs (again, drawcalls), you could maks them out and lerp the final material together.
Especially for things like lego, this would be optimal.