How big is TOO big for an open world on a simple dedicated server?

Well yes. using world composition and stuff you can keep your game at good rates. how you optimize the game is the real thing. also creating a open world map solo wont be easy.

also 5kmx5km is not a limit. there are games out there with map size about 40km2 and even 100 km2 +. one of the games that i really like is ark. but the game has a development team and obviously a experienced team behind. i would not recommend going too big and anywhere as close to this. just create a area that you can actually work on.