Shadow flicker with HWRT Renders -UE5.6

After days of testing I’ve narrowed down the cause of shadow flicker in Deferred renders to HWRT.
ie to fix I simply use r.Lumen.HardwareRayTracing 0

I’m using Anti Aliasing -None with 29 Spatial and 1 Temporal samples.
Since I’m rendering out Object ID passes

Is this a known bug?
Hoping for a fix since obviously Hardware Ray Tracing has lots of quality advantages.

What does the flicker look like?