Normal - Lumen causes shadow issues on meshes with default normal maps

I was trying to have a mesh with the default normal map and lumen causes weird shadows on it. Is there a way to not have it do that without turning it off?