I have a pretty large scene with 4k terrain and several thousend of foliage instances.
Because of the amount I must use Dynamic lighting, as lightmaps run out of memory.
My problem is that when looking towards the sun, I get 30fps, but when I look with the sun
it goes down to 15fps. I used different sun angles and the result is the same no matter where
the sun comes from and no matter how many objects are seen in the actual frame. the problem disapears
as soon as I turn of dynamic shadows. I also see this behaviour when I delete the foliage and only the terrain is left.
So it seems to be a general issue with dynamic shadows ?
It might be using a higher level of detail for the trees when you turn the camera if those trees are closer
Those trees have no LOD at the moment, just billboards. It also happens when I delete the foliage completely. It´s just less obvious. I found kind of a way around it by using distance field shadows. While they are much nicer and faster shadows I now have flickering in the far distance. But it´s more or less ok, now.
Regarding the distance field shadows. The flickering gets stronger and happens closer to the camera as further I set the distance. Ideally I would set the the distance so far the shadows cover my whole scene. I can´t do that as the flickering gets much to strong. If I would now rescale the whole scene to 50% I could set a smaller distance and the flickering would be nearly gone while it covers my whole scene. Is there maybe some sort of console command to set something like a initial scale for distance field shadows ? 
Have you tried using raytraced distance field shadows? They might offer a decent performance advantage.
As far as why does the performance get worse when looking along the light? It is most likely because in that situation, all the geometry “behind” the camera still has to render a shadow that is rendered on the trees in view. But when the light is going the other direction, the shadows cast out of view are casting onto objects that are also out of view so it does not cause as many additional draw calls.
Yes I´m experimenting with Distance Field Shadows. Actually it feels a bit like science fiction
I´m sitting here having 115k Instances in realtime, with some sort of raytraced shadows. All in view
and shadows until to the last tree. Unreal is probabely the right word.
Is there any way to increase the Distance field shadow quality with console commands that are not exposed in the gui ?