In this letter, we propose a robust and fast tracking framework by combining local and global appearance models to cope with partial occlusion and pose variations. The global appearance model is represented by a correlation filter to efficiently estimate the movement of the target and the local appearance model is represented by local feature points to handle partial occlusion and scale variations. Then global and local appearance models are unified via the Bayesian inference in our tracking framework. We experimentally demonstrate the effectiveness of the proposed method in both terms of accuracy and time complexity, which takes 12ms per frame on average for benchmark datasets.