In this paper, we propose a novel method for road sign detection and recognition in complex scene real world images. Our algorithm consists of four basic steps. First, we employ a regional contrast based bottom-up visual saliency method to highlight the traffic sign regions, which usually have dominant color contrast against the background. Second, each type of traffic sign has special color distribution, which can be explored by top-down visual saliency to enhance the detection precision and to classify traffic signs into different categories. A bag-of-words (BoW) model and a color name descriptor are employed to compute the special-class distribution. Third, the candidate road sign blobs are extracted from the final saliency map, which are generated by combining the bottom-up and the top-down saliency maps. Last, the color and shape cues are fused in the BoW model to express blobs, and a support vector machine is employed to recognize road signs. Experiments on real world images show a high success rate and a low false hit rate and demonstrate that the proposed framework is applicable to prohibition, warning and obligation signs. Additionally, our method can be applied to achromatic signs without extra processing.