Cant really say much about the performance drop from 4 to 5 since my performance opt. and debugging skills are not great. But for you it might be VSMs causing the issue. Try setting most of the scene to static validation (actor settings/rendering). You could also try changing the default lod bias for lumen shadows to a higher value. The default is -1.5 and it used to be 0 some engine versions ago.