This is something I believe is only happening since changing project from 4.22 to 4.23. The fps is dropping by up to 20 depending on if the camera is facing the same direction as the sunlight. Tested by rotating the light and seeing fps drop change accordingly.
Watch the FPS counter in the following video : Direction of light dropping FPS - YouTube
@Dokukai I’d say it’s not a fair test unless you do it with nothing in the level 
How do we know it’s not your rocks, for instance?.. ( or maybe just grass in the level ).
When camera is facing away from directional light, more objects are rendered into dynamic shadowmaps, hence, that is expected.
I’m getting this exact same bug actually. It seemed to happen in our project not simply after we upgraded it from 4.22 (or we simply didn’t notice somehow for a few weeks) but in the last week when we edited the intensity on the directional light. Now whenever you look in the general vicinity of where it’s pointing, you lose a significant amount of performance rendering the level unplayable. Deleting the directional light also doesn’t help, despite being deleted it still shows up in the scene rendering and tanks performance.