For whatever its worth I mostly notice this problem in areas with very low light, so you may be able to resolve it by just making your lights brighter and adjusting the exposure to compensate so that you can still continue to use the denoiser and lower sample counts.
With the denoiser disabled you’ll need to use far more samples to get a clean result.
Also probably of no use to you but this denoising artifact has been significantly improved in 5.1.