This paper describes a vision system with dual viewing angles, i. e., wide and narrow viewing angles, and a scheme of user-friendly speech dialogue environment based on the vision system. The wide viewing angle provides a wide viewing field for wide range motion tracking, and the narrow viewing angle is capable of following a target in wide viewing field to take the image of the target with sufficient resolution. For a fast and robust motion tracking, modified motion energy (MME) and existence energy (EE) are defined to detect the motion of the target and extract the motion region at the same time. Instead of using a physical device such as a foot switch commonly used in speech dialogue systems, the begin/end of an utterance is detected from the movement of user's mouth in our system. Without recognizing the movement of lips directly, the shape variation of the region between lips is tracked for more stable recognition of the span of a dialogue. The tracking speed is about 10 frames/sec when no recognition is performed and about 5 frames/sec when both tracking and recognition are performed without using any special hardware.

This paper describes the linguistic procedure of our speech dialogue system. The procedure is composed of two processes, syntactic analysis using a finite state network, and discourse analysis using a plan recognition model. The finite state network is compiled from regular grammar. The regular grammar is described in order to accept sentences with various styles, for example ellipsis and inversion. The regular grammar is automatically generated from the skeleton of the grammar. The discourse analysis module understands the utterance, generates the next question for users and also predicts words which will be in the next utterance. For an extension number guidance task, we obtained correct recognition results for 93% of input sentences without word prediction and for 98% if prediction results include proper words.

This paper describes design issues of a speech dialogue system, the evaluation of the system, and the data collection of spontaneous speech in a transportation guidance domain. As it is difficult to collect spontaneous speech and to use a real system for the collection and evaluation, the phenomena related with dialogues have not been quantitatively clarified yet. The authors constructed a speech dialogue system which operates in almost real time, with acceptable recognition accuracy and flexible dialogue control. The system was used for spontaneous speech collection in a transportation guidance domain. The system performance evaluated in the domain is the understanding rate of 84.2% for the utterances within the predefined grammar and the lexicon. Also some statistics of the spontaneous speech collected are given.

Progress of speech recognition based on the hidden Markov model has made it possible to realize man-machine dialogue systems capable of operating in real time. In spite of considerable effort, however, few systems have been successfully developed because of the lack of appropriate dialogue models. This paper reports on some of technology necessary to develop a dialogue system with which one can converse comfortably. The emphasis is placed on the following three points: how a human converses with a machine; how errors of speech recognition can be recovered through conversation; and what it means for a machine to be cooperative. We examine the first problem by investigating dialogues between human speakers, and dialogues between a human speaker and a simulated machine. As a consideration in the design of dialogue control, we discuss the relation between efficiency and cooperativeness of dialogue, the method for confirming what the machine has recognized, and dynamic adaptation of the machine. Thirdly, we review the research on the friendliness of a natural language interface, mainly concerning the exchange of initiative, corrective and suggestive answers, and indirect questions. Lastly, we describe briefly the current state of the art in speech recognition and synthesis, and suggest what should be done for acceptance of spontaneous speech and production of a voice suitable to the output of a dialogue system.