This paper presents a vision-based human interface system that enables a user to move a target object in a 3D CG world by moving his hand. The system can interpret hand motions both in a frame fixed in the world and a frame attached to the user. If the latter is chosen, the user can move the object forward by moving his hand forward even if he has changed his body position. In addition, the user does not have to keep in mind that his hand is in the camera field of view. The active camera system tracks the user to keep him in its field of view. Moreover, the system does not need any camera calibration. The key for the realization of the system with such features is vision algorithms based on the multiple view affine invariance theory. We demon-strate an experimental system as well as the vision algorithms. Human operation experiments show the usefulness of the system.