A flat normal map should be 0.5, 0.5, 1 rather than 0, 0, 1. Perhaps that is contributing to the artifacts you are seeing?