OK, so the distance to nearest works, but it doesn’t work exactly the way you think it should. If there’s a low-lying sand bar, those will still show up in the shore mask, even though they don’t perturb from the water at all. And if you want a longer distance, you will get waves skewing in all directions. What you need for a good shore mask is a hard edge where the water meets the surface, and a smooth gradient offset by a set distance equally around the shoreline. Distance to nearest is a smoother function that catches all sorts of objects you don’t want it to. If you want to perform this function procedurally, you could use the depth-based method and then possibly some fancy sobel filtering to expand it to a shore, but that will cost so much expense and the results really won’t be good at all because shores will be detected wherever objects are.
It seems using giant textures will be the best method for determining shorelines for a good long time.