I know that UE4 can do maps that are quite large (100 miles^2 or so). But I think that’s for open worlds like was used in the kite demo. How about doing a small city which would have tons of objects in it? It would seem the lighting demands would make this near impossible for UE4. Is there a way to make this work in a multiplayer environment?
A large city is doable, depending on the camera viewpoint. Keep in mind for FPS character viewpoint, a lot of culling will be done, so that most of the objects won’t be displayed. You will want to do lots of optimizations specific to your game, of course. If you want a top down viewpoint, with number of objects the similar to the Kite Demo, then you might run into issues, but keep in mind that the Kite Demo was had a huge number of objects in the scene as well, so it is doable.
Also, keep in mind that Batman: Arkham Knight was done in a modified Unreal 3, and it had a huge, destructible city. It is quite playable. And while UE3 and UE4 are not the same codebase, you have access the the UE4 codebase to make the changes you would need to achieve the same or better than they did.
Now, multiplayer does add more headaches, especially since you haven’t actually stated what type of game you want to create. A turn based game shouldn’t have any problems, nor should something along the lines of Farmville, where you could do the multiplayer aspect in a database, and not have to use the UE4 Multiplayer mode. Keep in mind something the size of the Kite Demo required lots of memory, 24GB in fact, which doesn’t play well in the server environment. You would have to come up with a strategy to compensate for that.
Unfortunately, I haven’t been involved in a multiplayer game project, so thats about as far as my knowledge goes. There are some great discussions in the forum about multiplayer optimizations, and requirements, so do a search, find someone that seems to know what they are talking about, and send them a PM. They may not reply, but they might, especially if they are EPIC Employees.
I was thinking along the lines of a multiplayer FPS, perhaps breaking the game map into tiles making it easier to create. But I’m still not sure how the lighting would work. I guess that if the map was mostly static meshes on top of a terrain, you would eat a lot of the video ram storing the light maps.
For large maps you will use dynamic lights in most of the cases, because it would take ages to build the light 
Except it would be much and much more performance friendly if you were to have static lights.
Sure, seven years from now when the build finishes
The enormous build time would be a very real concern. I could just see finding a small mistake that needed fixing and having to rebuild everything over again. On the other hand, asking the dynamic lighting to deal with something that large is asking a lot from the engine.
That’s where not rendering at production quality right away comes in. And baking using the cloud is a thing now too I believe.