This paper proposes a novel image classification scheme for cloth pattern recognition. The rotation and scale invariant delta-HOG (DHOG)-based descriptor and the entire recognition process using random ferns with this descriptor are proposed independent from pose and scale changes. These methods consider maximun orientation and various radii of a circular patch window for fast and efficient classification even when cloth patches are rotated and the scale is changed. It exhibits good performance in cloth pattern recognition experiments. It found a greater number of similar cloth patches than dense-SIFT in 20 tests out of a total of 36 query tests. In addition, the proposed method is much faster than dense-SIFT in both training and testing; its time consumption is decreased by 57.7% in training and 41.4% in testing. The proposed method, therefore, is expected to contribute to real-time cloth searching service applications that update vast numbers of cloth images posted on the Internet.