Hey FacePalm,
If you would create another video with the same scene and go to view show > visualize > Mesh Distance fields and record if it is flickering there.
Also if you could show the AO Shadows.
Lastly what is your Mesh Distance Field set to. If those buildings are outside of that range then the computation by default switches back to cascading shadows that have a limited range on them.
Thanks,