How to get rid of specular reflection spot without re-baking light?

For closed areas, a sphere reflection capture can be useful to get rid of such specular spots, but for open areas I’m still looking for a way to fix that.