Vanishing point estimation is an important issue for vision based road detection, especially in unstructured roads. However, most of the existing methods suffer from the long calculating time. This paper focuses on improving the efficiency of vanishing point estimation by using a heuristic voting method based on particle swarm optimization (PSO). Experiments prove that with our proposed method, the efficiency of vanishing point estimation is significantly improved with almost no loss in accuracy. Moreover, for sequenced images, this method is further improved and can get even better performance, by making full use of inter-frame information to optimize the performance of PSO.