Left and right orc/troll are trying to get to the middle and they jitter as a result. Does anyone know how to fix this issue? What could be the problem?

Personally, i wouldn´t even bother with collisions for formations. I would bring in something, where the general positions for different formations are stored (or build a formation manager), then you would only need to know, how many troops are in your current group/formation, and they then just need to figure out, which number they are in the choosen formation and walk to that position, relative to the current formation leader and his direction.

Then you can even give special units (like healer) special places in that formation (like center, surrounded by grunts), and they will choose the position depending on their function and number.

Aaand if you have several groups marching in formations or legions, then i bet, the army already have some schematics, where the leader of all those smaller groups should be (followed by their groups) in that bigger marching order ^.^