In case anyone else finds this post: I ran into the same problem and have a solution! While I don’t know if it’ll work for everyone, I hope it at least leads people down the right path.
I had this issue with the buffer stuck at two transactions, and Map Check was coming up with a couple of glaring issues. First, I had a lot of BSP at the time, and there were ~350 brushes that needed to be cleaned. So I ran “Clean Geometry Materials” (found in the Details tab of any brush). I also had a mesh with a null reference and deleted it. While I don’t know if both of those together fixed it, or if it was just one of them, but after that my Undo buffer was back to normal. It seems that there was a lot of excess memory usage which prevented a larger Undo buffer.
tl;dr Run a Map Check (under Build), fix as many issues as you can, and that will hopefully fix your Undo buffer.