How to get rid of specular reflection spot without re-baking light?

try disabling by features in the view panel and see what is causing that spot ! then post the results you can also use the viewmodes .