As it usually happens I found another solution to the problem as I was looking for something else
https://static.xx.fbcdn.net/images/emoji.php/v9/t4c/1/16/1f642.png
There is another denoiser that I did not know about: r.DiffuseIndirect.Denoiser: 0
If I set this to 0 the shadows are much more accurate, but extremely noisy. It needs a RTGI Samples Per Pixel of 1024 minimum, which is to expensive even for Movie Render Que rendering.
So increasing the resolution and keeping this denoiser seems a good middle ground