How to get rid of specular reflection spot without re-baking light?

There is no such settings for dynamic directional lights. LightSourceAngle in project settings only works for static directional lights.