I had the displaced UVs hooked up to the texture sampler UVs input. The values -1,2,-1,2 for the UV boundaries did the trick though!