Hi, from my experience so far, most problematic for GPU performance are:
(-) Many shadow casting lights or shadows casted by meshes that don’t need it (e.g. grass casting high quality shadows). Unreal has different shadow techniques for different usecases (e.g. for small things like grass you could use contact shadows). For larger scale shadows the new method is virtual shadow maps (is more performance heavy though), the old methods would be cascade shadow maps (does not work performant for high poly meshes) and/or distance field shadows and capsule shadows. And now they also have megalights for handling lots of shadow casting lights, never tried that though but that may be the way to go in a couple of engine versions when it gets more stable.
(-) Lots of overdraw and transparency (dense foliage)
And what I did not know about in the beginning but what also needs to be handled:
(-) Large texture resolution. Previously I thought that larger textures only cost VRAM but not performance, but they seem to also cost quite a lot performance. From your second image it looks like you are using very high res textures, try to keep them as low as possible (e.g. 8192 textures for a small plant would likely look the exact same as a 512 texture). You can set a maximum texture resolution, use mip bias and also make use of texture groups.
(-) Distance culling and mesh merging. Occlusion culling costs quite a lot of performance, so you can’t put in lots of meshes without handling culling. If you don’t use nanite, then you can use CullDistanceVolumes. If you use nanite, then you would need to implement your own distance culling system (maybe world partition also works). And in cases where you can’t use distance culling (e.g. seeing the exterior of a house from a distance, there you can distance cull the interior but not the exterior), you would need to use some kind of hlod system where you merge meshes and replace them with a simplified version. Unreal engine has an hlod system for that build in (I couldn’t get it to work properly when I tried to use it a while back, so I ended up manually merging the meshes via Tools → MergeActors and creating a custom swapping logic). Same for particle systems, those also keep costing performance even if they are not visible on the other end of the map. And lights have a max draw distance, so you can cull those by that.
Generally it would also be helpful to design the level with distance culling in mind, so limiting view distances.
And you can also use RenderDoc to get an in depth view of GPU performance cost, I found it very helpful for learning what costs performance.
Those are the things I found problematic so far and how I handled them, there may be other options now.