The most troublesome problem in automated X-ray mask inspection is how to exactly determine the threshold level for extracting the pattern portions of each scanning electron microscopic (SEM) image. An exact determination is difficult because the histogram shows, in most cases, a complicated multi-modal pattern and the true threshold level often varies with each successive image. This paper presents a novel thresholding approach for segmenting SEM images of X-ray masks. In this approach, the shape of the histogram of each image is iteratively analyzed until a threshold value minimizing the cost of the correspondence with a reference histogram and satisfying criteria for determining thresholds is obtained. This new approach is used in an automated inspection system. When the input image resolution is set to 0.05µm/pixel, experiments confirm 0.1µm defects are unfailingly detected.