I have had it do weird things before. I use a program called Process Lasso to force some processes to maintain running at Normal priority and full CPU affinity sometimes (i also use it to force them to run at Below Normal or adjust cpu affinity when I need to use the computer more than I need the builder to complete fast), which can sometimes make huge differences. I’ve often caught windows or some other actor forcing processes to only run on 1 core, which is weird and slows things down a lot.
Even if you don’t use it like that, Process Lasso can give you a really good overview of what is happening.