Orthographic camera with low ortho width value - poor perfomance

I have a 8x8km map, and an orthographic camera. The problem i that the lower is ortho width, the poorer the perfomance becomes. For example, when ortho width is 800000cm, I have stable 120 fps. than at around 400000 it starts dropping, and at about 5000 I’m getting 20 fps. Is that how it’s supposed to be, or am I missing something here?