Strange dark spots when using hard normals

I am using a normal map generated from a heightmap for an ocean. As I am increasing the strength of the normals, dark or “negative” looking spots appear at certain parts of the material.
The appearance of them depends on the camera position. So if I dont move, just rotate the camera, the spots doesn’t change. But when I move the camera, the spots can appear or disappear or change their positions. How can I get rid of these while keeping hard normals for the rest of the surface?