I understand your confusion, but there are some key points of interest when changing settings that you should be aware of:
- Real-time Ray Tracing by hardware on NVidia cards with low amount of VRAM needs DLSS, which basically will render in lower resolution and scale up. You could say about the same on AMD cards.
- If you have Ray traced GI (Global Illumination) turned on and it is using RT by hardware it will take a huge cost for open environments, but will be ok for closed environments.
- Size of the textures you are using might be too high, which will require large amount I/O to load the textures which will reduce the overall frame rates, and those sizes will cost more on HW RT if you are not using DLSS.
I recommend to make sure when you are enabling Lumen with HW RT, that Global Illumination is not enable with HW RT, let Lumen handle the GI.
It would be interesting if you could share more about the scene in question to allow the forum audience to give you more hints on whats going on. 