In this paper, a new hierarchical block matching algorithm using mean and difference pyramids is presented. The detection of motion vectors at each level of the pyramid is accomplished by selectively eliminating the candidate motion vectors that cannot provide the best match at the next lower level. The remaining motion vectors at each level are propagated and used as the initial motion vectors at the next lower level. Therefore, the possibility of falling into local minima can be significantly reduced. The simulation results show that the proposed method has excellent performance with reduced computational complexity.