Heightmap shows buildings, how to clean the map?

I am using a real world height map, but it also shows the elevation of buildings, not only terrain. So I end up having roads, parcels, etc. in my 3d terrain that I don’t want there.

I want just the natural accidents, not buildings.

How could i clean the heightmap? I don’t want to have to modify everything in engine with the landscape tools since it would be too slow. Any better way? GIMP maybe to modify the heightmap?