Okay, found the issue. Weirdly, the HDRI I was using was causing the problem. After I changed it the light spot went away.