We describe a new system for high-speed wireless access systems between base stations and mobile terminals. In the proposed system, the base station has an array antenna and tracks mobile terminals by using a new tracking algorithm. A radio-on-fiber technique is used to simplify and miniaturize the components of the base station. Estimating the direction-of-arrival of the signals from a mobile terminal is important in implementing the proposed system. We propose a new tracking algorithm that uses directions-of-arrival, angular velocities of mobile terminals, and scatter modeling in multipath communications channels to improve the tracking performance. We also developed experimental equipment to demonstrate the feasibility of the proposed millimeter-wave broadband wireless access system and the efficiency of the tracking algorithm using an array antenna system. In this paper, we describe our system and present a new approach for tracking mobile terminals, which is the key feature of the system. We also discuss our simulation and experimental results.