Literally just putting an add node connected to specular and making it 0 + 0 or 0 + 0.1 fixed it. Thanks.
