This paper presents a simulated annealing (SA)-based algorithm for fast and robust block motion estimation. To reduce computational complexity, the existing fast search algorithms move iteratively toward the winning point based only on a finite set of checking points in every stage. Despite the efficiency of these algorithms, the search process is easily trapped into local minima, especially for high activity image sequences. To overcome this difficulty, the new algorithm uses two sets of checking points in every search stage and invokes the SA to choose the appropriate one. The employment of the SA provides the search a mechanism of being able to move out of local minima so that the new algorithm is less susceptible to such a dilemma. In addition, two schemes are employed to further enhance the performance of the algorithm. First, a set of initial checking points which exploit high correlations among the motion vectors of the temporally and spatially adjacent blocks are used. Second, an alternating search strategy is addressed to visit more points without increasing computations. Simulation results show that the new algorithm offers superior performance with lower computational complexity compared to previous works in various scenarios.