This paper presents a 3D feature-based binocular tracking algorithm for tracking crowded people indoors. The algorithm consists of a two stage 3D feature points grouping method and a robust 3D feature-based tracking method. The two stage 3D feature points grouping method can use kernel-based ISODATA method to detect people accurately even though the part or almost full occlusion occurs among people in surveillance area. The robust 3D feature-based Tracking method combines interacting multiple model (IMM) method with a cascade multiple feature data association method. The robust 3D feature-based tracking method not only manages the generation and disappearance of a trajectory, but also can deal with the interaction of people and track people maneuvering. Experimental results demonstrate the robustness and efficiency of the proposed framework. It is real-time and not sensitive to the variable frame to frame interval time. It also can deal with the occlusion of people and do well in those cases that people rotate and wriggle.