Object extraction and tracking in a video image is basic technology for many applications, such as video surveillance and robot vision. Many moving object extraction and tracking methods have been proposed. However, they fail when the scenes include illumination change or light reflection. For tracking the moving object robustly, we should consider not only the RGB values of input images but also the shape information of the objects. If the objects' shapes do not change suddenly, matching positions on the cost matrix of exclusive block matching are located nearly on a line. We propose a method for obtaining the correspondence of feature points by imposing a matching position constraint induced by the shape constancy. We demonstrate experimentally that the proposed method achieves robust tracking in various environments.