We have developed a map-based approach that enables us to efficiently extract information about man-made objects, such as buildings, from aerial images. An image is matched with a corresponding map in order to estimate the object information in the image (i. e. , presence, location, shape, size, kind, and surroundings). This approach is characterized by using a figure contained in a map as an object model for a top-down (model-driven) analysis of an object in the aerial image. We determined the principal steps of the map-based approach needed to extract object information and update a map. These steps were then applied to obtain the locations of missing buildings and the heights of existing buildings. The extraction results of experiments using aerial images of Kobe City (taken after the 1995 earthquake) show that the approach is effective for automatically extracting building information from aerial images and for rapidly updating map data.