Only raytraced shadows - Huge performance impact

Hi guys, having a question about the performance using ray tracing.
For an outdoor scene, I want just to use ray traced shadows.
To be sure, I deactivated everything, expect shadows, in the DefaultEngine.ini


The problem is, ray traced shadows are killing the fps.
Almost 80 fps.

For testing purpose, I deactivated the shadows with the command line and this restored the good performance.
So there is no other feature in the background, which is killing the frames (i think).
So I went to the only two lights in the scene (directional and skylight) and deactivated the ray traced shadows there. Visually they are gone, but the framefrate is still miserable.

Any ideas how to fix it?