Noise on Raytrace Shadow Edges with Soft lights

Hi, I’m trying to render some very smooth models (a sphere with some protrusions) that use simple textures: (flat single color albedo, .8 roughness). I am using raytraced shadows on rect lights and am noticing a lot of noise on the edges of the shadows. I can push the raytrace samples per pixel on the light to absurdly high numbers (1000) so that my viewport almost freezes and I still see noise.

I see the same noise on directional lights with a source angle above 1 (eg, when the shadows start to soften)

Any tips on reducing this noise would be greatly appreciated. When using models with normal/albedo/roughness textures it is still there but much less visible b/c of the visual noise of said textures. On a naked model with a simple texture it is very visible.

Having the same issues, trying to play around with RT and HDRI backdrops. My shadow edges are really noisy.