To track the players and shuttlecock in broadcast badminton video is a challenge, especially for tracking the small size and fast moving shuttlecock. There are many situations that may cause occlusion or misdetection. In this paper, a method is proposed to track players and shuttlecock in broadcast badminton videos. We apply adaptive Kalman filter, trajectory confidence estimation and confidence-update (Location Similarity and Relative Motion Relation, RMR) to improve the accuracy of object trajectories. In our experiments, the proposed method significantly enhance the tracking success rate of players and shuttlecock.