An automatic defect classification system (ADC) for use in visual inspection of semiconductor wafers is introduced. The methods of extracting the defect features based on the human experts' knowledge, with their correlations with the defect classes are elucidated. As for the classifier, Hyperellipsoid Clustering Network (HCN) which is a layered network model employing second order discrimination borders in the feature space, is introduced. In the experiments using a collection of defect images, the HCNs are compared with the conventional multilayer perceptron networks. There, it is shown that the HCN's adaptive hyperellipsoidal discrimination borders are more suited for the problem. Also, the cluster encapsulation by the hyperellipsoidal border enables to determine rejection classes, which is also desirable when the system will be in actual use. The HCN with rejection achieves, an overall classification rate of 75% with an error rate of 18%, which can be considered equivalent to those of the human experts.