Ok so you want tesselation only on some parts of the material? Hmm… I have honestly never tried that before, but as far as I’m aware the way the tutorial type of distance based tesselation works is using a sphere mask, which is essentially a black and white mask. If that works, than I don’t see how you wouldn’t be able to just use a mask to define which areas need tesselation and which ones don’t. Maybe give that a try (if you haven’t already?)
A mask is just a range of values between 0 and 1 (if it’s clamped anyway), 0 tesselation to 1 full tesselation. So basically a lerp should work.
Also what do you mean with “layers”. Do you mean like vertex painting? Because that could also work fine with a mask.