Advanced driver assistance system (ADAS) can recognize traffic signals, vehicles, pedestrians, and so on all over the vehicle. However, because the ADAS is based on images taken in an outdoor environment, it is susceptible to ambient weather such as fog. So, preprocessing such as de-fog and de-hazing techniques is required to prevent degradation of object recognition performance due to decreased visibility. But, if such a fog removal technique is applied in an environment where there is little or no fog, the visual quality may be deteriorated due to excessive contrast improvement. And in foggy road environments, typical fog removal algorithms suffer from color distortion. In this paper, we propose a temporal filter-based fog detection algorithm to selectively apply de-fogging method only in the presence of fog. We also propose a method to avoid color distortion by detecting the sky region and applying different methods to the sky region and the non-sky region. Experimental results show that in the actual images, the proposed algorithm shows an average of more than 97% fog detection accuracy, and improves subjective image quality of existing de-fogging algorithms. In addition, the proposed algorithm shows very fast computation time of less than 0.1ms per frame.