An automatic zip code recognition system for Japanese mail is proposed in this paper. It is assumed that a zip code is composed of three numerals and requited to be written in a specified frame. In actual images, however, the three numerals sometimes extend outside the specified frame and are not clearly separated. Considering this situation, the authors devised a system with two stages, the segmentation stage and the recognition stage. The segmentation stage consists of five steps: setting and adjusting of initial areas for numeral images (figures), calculation of the center of gravity of each figure, search for the horizontal and vertical boundaries of each figure, determination of the final area for each figure, and normalization of the figure in each final area. In the recognition stage, the Localized Arc Pattern Method (Arc method) proposed by Yoshimura et al. (1991) is implemented hierarchically; that is, a simple Arc method is applied first to each figure and a more complex one is applied subsequently unless the figure is identified in the first step. In the recognition process, every figure is judged as a numeral or otherwise rejected. The proposed system was applied to a database provided by the Institute for Post and Telecommunications Policy (IPTP). The segmentation algorithm yielded an adequate result. The recognition algorithm yielded scores as high as 90.6% in correct recognition rate and 0.7% in error rate. The best score of the precision index (P-index) specified by the IPTP was as low as 15.7 for the above mentioned IPTP database, while the score for another IPTP database was 16.9.