I found the answer and had a moment of deja vu. You need to turn parallel builds down to 1 in VS. You can possibly get away with more but most likely not.
This will cause it to build slower, but it will actually work. Seems like a crazy thing to not have fixed in 22 versions of the engine though.