In this study, we introduce a software pipeline to track feature points across endoscopic video frames. It deals with the common problems of low contrast and uneven illumination that afflict endoscopic imaging. In particular, irregular feature trajectories are eliminated to improve quality. The structure of soft tissue is determined by an iterative factorization method based on collection of tracked features. A shape updating mechanism is proposed in order to yield scale-invariant structures. Experimental results show that the tracking method produced good tracking performance and increased the number of tracked feature trajectories. The real scale and structure of the target scene was successfully estimated, and the recovered structure is more accuracy than the conventional method.