Template matching for image sequences captured with a moving camera is very important for several applications such as Robot Vision, SLAM, ITS, and video surveillance systems. However, it is difficult to realize accurate template matching using only visual feature information such as HSV histograms, edge histograms, HOG histograms, and SIFT features, because it is affected by several phenomena such as illumination change, viewpoint change, size change, and noise. In order to realize robust tracking, structure information such as the relative position of each part of the object should be considered. In this paper, we propose a method that considers both visual feature information and structure information. Experiments show that the proposed method realizes robust tracking and determine the relationships between object parts in the scenes and those in the template.