It has been 6 hrs no reboot.
I think either the GPU is the problem, AMD drivers or the PSU.
Right now 16 pins of PSU are not connected to any GPU, so PSU is drawing much less power. I will try getting a high-end 16 pin GPU to check whether GPU is the problem or PSU is not being able to take the load of GPU.