Of course you will get some additional shadowing from RTAO. RT can see outside of the camera frustum while SS can not. But the significant performance impact and denoising artifacts from RTAO aren’t worth the minor changes in the image.
Your materials are hardly accurate when your windows are made out of metal rather than dielectric material 
Lastly, if you don’t see any room to improve the quality of the building, at least on the texturing/shading level, then that’s your problem. But at this point, you are probably wasting your time digging too deep into AO and GI effects, because that’s just not the place where realism is missing in your scene.