Turning ray tracing off cuts my FPS in half?

As the title says - I turned ray tracing on and it doubled my fps but when I turn it off its cut in half, which surprised me as I thought it would be the other way round.

With it i’m getting really bad black holes in my textures and the shadows are nearly all gone. I’d rather not have it i’m just really curious to the heavy fps drop?

anyone have this issue or solution?

thanks in advance!