Exactly how much of the terrain details are being loaded into memory at any one given time? You have to set a limit especially in a multiplayer game over a network. You have to limit the distance from the player that high detail is being drawn and making this an option so players with a poor pc can scale it back to minimum but players with high end rigs can crank it right up is probably optimal. Regardless the distance and detail are most likely what is killing the framerate. You might be able to optimize some other areas but distance/detail and how much of the world you actually have loaded all matter an awful lot.
What kind of hardware are you currently running this on? My machine is crazy old and I can still get a decent rate in a lot of stuff. It might help if you posted the specs just for reference.