That is quite the visual, and I would check your characters camera as this could be where the problem is being introduced. Firstly, I would look at the Post Processing of the camera and make sure the values there are not overriding your scenes post process volume.
If the roughness of your material applied to these walls is a constant value of 1.0 then you should not be seeing any sort of shimmer.