Auto-calibration for structure and motion recovery can be used for match move where the goal is to insert synthetic 3D objects into real scenes and create views as if they were part of the real scene. However, most auto-calibration methods for multi-views utilize bundle adjustment with non-linear optimization, which requires a very good starting approximation. We propose a novel key-frame selection measurement and LMedS (Least Median of Square)-based approach to estimate scene structure and motion from image sequences captured with a hand-held camera. First, we select key-frames considering the ratio of number of correspondences and feature points, the homography error and the distribution of corresponding points in the image. Then, by using LMedS, we reject erroneous frames among the key-frames in absolute quadric estimation. Simulation results demonstrated that the proposed method can select suitable key-frames efficiently and achieve more precise camera pose estimation without non-linear optimization.