Looks like the issue is related to the shadow denoiser. If you disable it, the problem goes away.
r.Shadow.Denoiser 0