Hi, few years ago at Bohemia Interactive I was leading development of Seattle’s 60x60km map for TakeOn Helicopters.
I’d definitely go for OpenStreetMap data as you can extract vector data into any 3D modelling tool(via shp,dxf, obj… format). I don’t recommend using an aerial imagery, only if you have it in highres(10cm) and ortho rectified, otherwise it can be pretty inaccurate. Modelling a city or part of it also requires some knowledge of GIS principles and tools(like QuantumGIS,GlobalMapper,ARCGIS and so on), but there already are importers for OSM format to several 3D modelling tools, depends on what attributes(bulding height, roof type etc.) you need to extract from a dataset.
Regarding the elevation data, as someone already mentioned you can use USGS data, but as far as I remember data available there are still in 10m/px resolution which is a bit rough for urban areas to have nice details on the group, but depends on what detail you expect to have. You can use 10m/px resolution at least to place buildings on a terrain.