That is not expected. First, check the Lumen Surface Cache viewmode to make sure those areas have Surface Cache coverage. If those walls have coverage, then it is most likely the issue described here.
And also it “ghosts” all background as they were illuminated by the light source
This is caused by Screen Traces picking up translucency in the foreground. We don’t currently have a solution but will be working on it in the future.
Also in the same scene, I got a drop of 50% of the FPS with Lumen enabled:
You should expect an fps drop when enabling dynamic GI, it costs something. If you drop down the Global Illumination scalability to Medium, Lumen GI will be disabled, for GPU’s that can’t afford it.