That’s called specular aliasing. It happens when you have glossy surfaces with high frequency normal information and move away far enough that the normal variations get blended together, resulting in strong peaks in the reflections.
One solution is to modulate the roughness texture mipmaps using the normal map data, so that the increased roughness further away tones down the highlights:
This presentation has some nice graphics on the issue:
http://alex.vlachos.com/graphics/Alex_Vlachos_Advanced_VR_Rendering_GDC2015.pdf