This paper presents the design of an automatic surveillance system to monitor the dangerous non-frontal gazes of the car driver. To track the driver's eyes, we propose a novel filter to locate the "between-eye", which is the middle point between the two eyes, to help the fast locating of eyes. We also propose a specially designed criterion function named mean ratio function to accurately locate the positions of eyes. To analyze the gazes of the driver, a multilayer perceptron neural network is trained to examine whether the driver is losing the proper gaze or not. By incorporating the neural network output with some well-designed alarm-issuing rules, the system performs the monitoring task for single dedicated driver and multiple different drivers with a satisfied performance in our experiments.