Three-dimensional (3-D) scattering center models use a finite number of point scatterers to efficiently represent complex radar target signature. Using the CLEAN algorithm, 3-D scattering center model is extracted from the inverse synthetic aperture radar (ISAR) image, which is generated based on the shooting and bouncing ray (SBR) technique. The conventional CLEAN extracts the strongest peak iteratively based on the assumption that the scattering centers are isolated. In a realistic target, however, both interference from the closely spaced points and additive noise distort the extraction process. This paper proposes a matched filter-based CLEAN algorithm to improve accuracy efficiently. Using the matched filtering of which impulse response is the known point spread function (PSF), a point most correlated with the PSF is extracted. Thus, the proposed method optimally enhances the accuracy in the presence of massive distortions. Numerical simulations using canonical and realistic targets demonstrate that the extraction accuracy is improved without loss of time-efficiency compared with the existing CLEAN algorithms.