This paper describes an adaptive feature extraction method that exploits category-specific information to overcome both image degradation and deformation in character recognition. When recognizing multiple fonts, geometric features such as directional information of strokes are often used but they are weak against the deformation and degradation that appear in videos or natural scenes. To tackle these problems, the proposed method estimates the degree of deformation and degradation of an input pattern by comparing the input pattern and the template of each category as category-specific information. This estimation enables us to compensate the aspect ratio associated with shape and the degradation in feature values and so obtain higher recognition accuracy. Recognition experiments using characters extracted from videos show that the proposed method is superior to the conventional alternatives in resisting deformation and degradation.