Reasons for getting double walls?

I have collected images of a large building. All the images were collected from the ground. I was very careful to get plenty of overlap. The lighting conditions were bright but overcast so there are no shadows. The images are well focused with with a shutter speed fast enough to not worry about too much motion blur. The camera was Sony Next-5T with a 30mm prime lens, 45mm equivalent.

Most of the images aligned, but there are 2 walls that appear to be getting created twice. I highlighted the problem areas in red in the screen capture below.One set of the double walls run parallel with each other, while the other one slowly diverge from the building’s corner.

I have read many of the posts on the forum and have tried grouping the images wrt exif, and selecting other camera models. Everything I try has many small changes, but nothing really appears to be making an impact on the double wall affect. 

I can go back and get more images if that could help, but I really want to understand the why this is happening. Any advice or shared knowledge would be very much appreciated.

Its called ghosting. I usually get it on repeating patterns. Control points on a few of the misaligned pictures usually does the trick.

Thanks for the great info. Adding a few control points did the trick!

Also for future have a look at the image above, Note the number of images along the face of each wall compared to corners. you need more images as you go around the corners this is likely what caused the data to split