An FPGA-based hardware hand sign recognition system was proposed in our previous work. The hand sign recognition system consisted of a preprocessing and a self-organizing map (SOM)-Hebb classifier. The training of the SOM-Hebb classifier was carried out by an off-chip computer using training vectors given by the system. The recognition performance was reportedly improved by adding perturbation to the training data. The perturbation was added manually during the process of image capture. This paper proposes a new off-chip training method with automatic performance improvement. To improve the system's recognition performance, the off-chip training system adds artificially generated perturbation to the training feature vectors. Advantage of the proposed method compared to additive scale perturbation to image is its low computational cost because the number of feature vector elements is much less than that of pixels contained in image. The feasibility of the proposed off-chip training was tested in simulations and experiments using American sign language (ASL). Simulation results showed that the proposed perturbation computation alters the feature vector so that it is same as the one obtained by a scaled image. Experimental results revealed that the proposed off-chip training improved the recognition accuracy from 78.9% to 94.3%.