I find it confusing that culling is coming into it when the walls should be blocking objects beyond the walls.
It is true that the objects are already being not rendered because they are not seen, but there are various ways that the engine figures out how they aren’t seen in the first place. The distance pass is one of the most performant parts of deciding what is/isn’t on the screen at the time. Otherwise, you have to rely on frustum culling or occlusion culling for it to figure out that the objects aren’t on the screen at the time, and those can be more expensive than distance culling.
Though, really, culling was just a guess that didn’t end up mattering much anyway because according to your stat GPU, the problem is clearly the shadows!
How many lights in your scene are movable or otherwise dynamic? Do you have any large dynamic lighting features enabled?
EDIT: You could also try baking all your static lighting if you haven’t yet. I don’t know if unbaked lighting previews would cause anything, but we are trying everything we can think of here. 
2nd EDIT: I have an idea on how to test if it is 100% just the shadows or if it is other things. Go to settings (from the viewport toolbar) then to engine scalability settings and set shadows to low. This is just the shadows while running the engine (it won’t affect a finished game), but it is useful to test if that is what you need to fix or not.