Yeah you may have something there, thanks I appreciate it.
Yeah the area on the right side with the top of the ladder showing because of how close the player is to wall, is the tricky one. see if the character closer to the ladder next to it then the view to the ladder above get blackened. Also the top and below the stairs is another tricky one.
EDIT: also the blue wall culling needs to depend on the angle of view, so that it works properly for the rest