Yes, you can do something like this
You also have to do exactly the same for the normals, otherwise it just look like a big mash or rubbish
Or you can control the blend amount with a parameter
Or different noise textures
