A new algorithm for fast detection and tracking of moving targets using a mobile video camera is presented. Our algorithm is based on image feature detection and matching. To detect features, we used edge points and their accumulated curvature. When the features are detected they are matched with their corresponding points using a new method called fuzzy-edge based feature matching. The proposed algorithm has two modes: detection and tracking. In the detection mode, background motion is estimated and compensated using an affine transformation. The resultant motion-rectified image is used for detection of the target location using split and merge algorithm. We also checked other features for precise detection of the target. When the target is identified, algorithm switches to the tracking mode, which also has two phases. In the first phase, the algorithm tracks the target with the intention to recover the target bounding-box more precisely and when the target bounding-box is determined precisely, the second phase of tracking algorithm starts to track the specified target more accurately. The algorithm has good performance in the environment with noise and illumination change.