It must be a boundingscale issue. I have no idea what else might cause this. If you’re willing to go to some more effort, try implementing distance based tesselation.
You can then essentially just kill displacement beyond anything but close-ups. If that doesn’t take care of your issue I’ll eat my shoe.