yep. sometimes it do that.
I was having issues with that a few days ago, and I popped open task manager, and noticed that i had 16 cl.exe processes running but they were all almost 0 cpu usage. I pulled out another utility I use called Process Lasso, and it told me that all 16 of the cl.exe processes were for some reason running on a single core at Below Normal priority.
I used that app to set them to run on all 16 cores at normal priority , and it stopped doing that delay thing.