Recently, Intelligent Transport Systems (ITS) are being researched and developed briskly. As a part of ITS, detecting vehicles from images taken by a camera loaded on a vehicle are conducted. From such backgrounds, authors have been conducting vehicle detection in nighttime. To evaluate the accuracy of this detection, gold standards of the detection are required. At present, gold standards are created manually, but manually detecting vehicles take time. Accordingly, a system which detects vehicles accurately without human help is needed to evaluate the accuracy of the vehicle detection in real time. Therefore the purpose of this study is to automatically detect vehicles in nighttime images, taken by an in-vehicle camera, with high accuracy in offline processing. To detect vehicles we focused on the brightness of the headlights and taillights, because it is difficult to detect vehicles from their shape in nighttime driving scenes. The method we propose uses Center Surround Extremas, called CenSurE for short, to detect blobs. CenSurE is a method that uses the difference in brightness between the lights and the surroundings. However, blobs obtained by CenSurE will also include objects other than headlights and taillights. For example, streetlights and delineators would be detected. To distinguish such blobs, they are tracked in inverse time and vehicles are detected using tags based on the characteristics of each object. Although every object appears from the same point in forward time process, each object appears from different places in images in inverse time processing, allowing it to track and tag blobs easily. To evaluate the effectiveness of this proposed method, experiment of detecting vehicles was conducted using nighttime driving scenes taken by a camera loaded on a vehicle. Experimental results of the proposed method were nearly equivalent to manual detection.