That sounds somewhat expected. “Occlusion Culling” works out we can avoid drawing, if another object is infront of it. This is a fairly expensive process, often counter-productive, but in a large scene with lots of “walls” (or perhaps mountains) preventing you from seeing most of the secne at any one time, it may help more.
Disabling it will spend less time working out what to draw, and more time drawing, as there is more to draw.