Can vector displacement be applyed through a mask?

I made a vector displacement map for a character that makes it muscular. However I want to be able to apply that with a mask so that only the arm or leg or midsection get that map. The problem is that when I do it I get tearing in the transition area. Is there a way to do this?