In this letter, we propose a novel discriminative representation for patterned fabric defect inspection when only limited negative samples are available. Fisher criterion is introduced into the loss function of deep learning, which can guide the learning direction of deep networks and make the extracted features more discriminating. A deep neural network constructed from the encoder part of trained autoencoders is utilized to classify each pixel in the images into defective or defectless categories, using as context a patch centered on the pixel. Sequentially the confidence map is processed by median filtering and binary thresholding, and then the defect areas are located. Experimental results demonstrate that our method achieves state-of-the-art performance on the benchmark fabric images.