How do I get a single texture to use displacement?

I am trying to get a single texture to use displacement but every time I try to use it the way the live tutorial has explained it every texture in that scene gets displaced based on that one textures height map. How do I make this so it just affects a single texture for the landscape when painting?