We are developing multimodal man-machine interfaces through which users can communicate by integrating speech, gaze, facial expressions, and gestures such as nodding and finger pointing. Such multimodal interfaces are expected to provide more flexible, natural and productive communications between humans and computers. To achieve this goal, we have taken the approach of modeling human behavior in the context of ordinary face-to-face conversations. As the first step, we have implemented a system which utilizes video and audio recording equipment to capture verbal and nonverbal information in interpersonal communications. Using this system, we have collected data from a task-oriented conversation between a guest (subject) and a receptionist at company reception desk, and quantitatively analyzed this data with respect to multi-modalities which would be functional in fluid interactions. This paper presents detailed analyses of the data collected: (1) head nodding and eye-contact are related to the beginning and end of speaking turns, acting to supplement speech information; (2) listener responses occur after an average of 0.35 sec. from the receptionist's utterance of a keyword, and turn-taking for tag-questions occurs after an average of 0.44 sec.; and (3) there is a rhythmical coordination between speakers and listeners.

We developed an eye-contact technique using a blazed half-transparent mirror (BHM), which is a micro-HM array arranged on the display surface, to make a compact eye-contact videophone. This paper describes a new BHM structure that eliminates ghosts and improves image quality. In the new BHM, the reflection and transmission areas are separated to exclude ghosts from appearing in the captured image. We evaluated the characteristics of the captured and displayed images. The results show that the contrast ratio of the captured image and the brightness of both captured and displayed images are much better than with the previous BHM.

An eye-contact technique using a blazed half-transparent mirror (BHM) is developed. This half-transparent mirror (HM) consists of an in-line array of many slanting micro-HMs. We fabricated a prototype system and confirmed the principle of this technique. The resolution of an image reflected by a BHM was simulated to determine how to improve the image quality and the factors degrading the resolution were clarified.