Strange detail lighting issue

I have a fridge in my scene with a brushed metal material on it and its getting really blown out for some reason when I build my lighting. Has anyone else had this issue? I don’t have the roughness map turned up extremely high or anything, it just gets blown out when the lighting gets built. I have the lightmap res set to 256 as well.