The processor. It’s an old 4 core processor so don’t expect anything from it. The other important factors are RAM and an SSD. The video card is irrelevant in compile times. So if you want to increase compile times on a budget, I recommend upgrading the latter ones.
I had a similar spec before, and I had around 8-10 min build times. After upgrading my PC compile times were reduces to just 8-10 seconds! It’s crazy lol. (Ryzen 7 5800X, 32 GB RAM, M.2 NVME SSD)