That’s normal in my eyes. Tiny errors add up.
Also, my equipment is not the fanciest.
But most importantly, those are very difficult circumstances, basically one plane of masonry consisting of several “arms” only connected in the middle. If you have a whole building that you can circle completely, then the errors go down significantly automatically.
Once I followed Vladlens workflow outlined here How to optimize Reconstruction (use lens grouping by exif), the result was almost perfect.
I used something like 30 odd GCPs obtained by theodolite and the worst deviation was 3 (in words: three !) cm in the bottom right corner. That is very well within acceptable tolerances already. But with another alignment with the GCPs present, the error went down to a few mm.
Satisfied? 