A user wearing a Head-Mounted Display (HMD) is likely to feel isolated when sharing virtual reality (VR) experiences with Non-HMD users in the same physical space due to not being able to see the real space outside the virtual world. This research proposes a method for an HMD user to recognize the Non-HMD users' gaze and attention via a projector attached to the HMD. In the proposed approach, the projected HMD user's view is filtered darker than default, and when Non-HMD users point controllers towards the projected view, the filter is removed from a circular area for both HMD and Non-HMD users indicating which region the Non-HMD users are viewing. We conducted two user studies showing that the Non-HMD users' gaze can be recognized with the proposed method, and investigated the preferred range for the alpha value and the size of the area for removing the filter for the HMD user.

Gaze following is the task of estimating where an observer is looking inside a scene. Both the observer and scene information must be learned to determine the gaze directions and gaze points. Recently, many existing works have only focused on scenes or observers. In contrast, revealed frameworks for gaze following are limited. In this paper, a gaze following method using a hybrid transformer is proposed. Based on the conventional method (GazeFollow), we conduct three developments. First, a hybrid transformer is applied for learning head images and gaze positions. Second, the pinball loss function is utilized to control the gaze point error. Finally, a novel ReLU layer with the reborn mechanism (reborn ReLU) is conducted to replace traditional ReLU layers in different network stages. To test the performance of our developments, we train our developed framework with the DL Gaze dataset and evaluate the model on our collected set. Through our experimental results, it can be proven that our framework can achieve outperformance over our referred methods.

A method to generate color palettes from images is proposed. Here, deep neural networks (DNN) are utilized in order to consider human perception. Two aspects of human perception are considered; one is attention to image, and the other is human preference for colors. This method first extracts N regions with dominant color categories from the image considering human attention. Here, N is the number of colors in a color palette. Then, the representative color is obtained from each region considering the human preference for color. Two deep neural-net systems are adopted here, one is for estimating the image area which attracts human attention, and the other is for estimating human preferable colors from image regions to obtain representative colors. The former is trained with target images obtained by an eye tracker, and the latter is trained with dataset of color selection by human. Objective and subjective evaluation is performed to show high performance of the proposed system compared with conventional methods.

We propose an attention mechanism in deep learning networks for gender recognition using the gaze distribution of human observers when they judge the gender of people in pedestrian images. Prevalent attention mechanisms spatially compute the correlation among values of all cells in an input feature map to calculate attention weights. If a large bias in the background of pedestrian images (e.g., test samples and training samples containing different backgrounds) is present, the attention weights learned using the prevalent attention mechanisms are affected by the bias, which in turn reduces the accuracy of gender recognition. To avoid this problem, we incorporate an attention mechanism called gaze-guided self-attention (GSA) that is inspired by human visual attention. Our method assigns spatially suitable attention weights to each input feature map using the gaze distribution of human observers. In particular, GSA yields promising results even when using training samples with the background bias. The results of experiments on publicly available datasets confirm that our GSA, using the gaze distribution, is more accurate in gender recognition than currently available attention-based methods in the case of background bias between training and test samples.

We investigate an interactive evolutionary computation (IEC) using multiple users' gaze information when users partially participate in each design evaluation. Many previous IEC systems have a problem that user evaluation loads are too large. Hence, we proposed to employ user gaze information for evaluating designs generated by IEC systems in order to solve this problem. In this proposed system, users just view the presented designs, not assess, then the system automatically creates users' favorite designs. With the user's gaze information, the proposed system generates coordination that can satisfy many users. In our previous study, we verified the effectiveness of the proposed system from a real system operation viewpoint. However, we did not consider the fluctuation of the users during a solution candidate evaluation. In the actual operation of the proposed system, users may change during the process due to the user interchange. Therefore, in this study, we verify the effectiveness of the proposed system when varying the users participating in each evaluation for each generation. In the experiment, we employ two types of situations as assumed in real environments. The first situation changes the number of users evaluating the designs for each generation. The second situation employs various users from the predefined population to evaluate the designs for each generation. From the experimental results in the first situation, we confirm that, despite the change in the number of users during the solution candidate evaluation, the proposed system can generate coordination to satisfy many users. Also, from the results in the second situation, we verify that the proposed system can also generate coordination which both users who participate in the coordination evaluation can more satisfy.

We paid attention the amount of change for each resolution by specifying the gaze position of images, and measured accommodation and convergence eye movement when watching high-resolution images. Change of convergence angle and accommodation were like the actual depth composition in the image when images were presented in the high-resolution.

An eye model expressed by a revolution about the optical axis of the eye is one of the most accurate models for use in a 3D gaze estimation method. The measurement range of the previous gaze estimation method that uses two cameras based on the eye model is limited by the larger of the two angles between the gaze and the optical axes of two cameras. The previous method cannot calculate the gaze when exceeding a certain limit of the rotation angle of the eye. In this paper, we show the characteristics of reflections on the surface of the eye from two light sources, when the eye rotates. Then, we propose a method that extends the rotation angle of the eye for a 3D gaze estimation based on this model. The proposed method uses reflections that were not used in the previous method. We developed an experimental gaze tracking system for a wide projector screen and experimentally validated the proposed method with 20 participants. The result shows that the proposed method can measure the gaze of more number of people with increased accuracy compared with the previous method.

We propose a novel framework for integrating beginners' machine operational experiences with those of experts' to obtain a detailed task model. Beginners can provide valuable information for operation guidance and task design; for example, from the operations that are easy or difficult for them, the mistakes they make, and the strategy they tend to choose. However, beginners' experiences often vary widely and are difficult to integrate directly. Thus, we consider an operational experience as a sequence of hand-machine interactions at hotspots. Then, a few experts' experiences and a sufficient number of beginners' experiences are unified using two aggregation steps that align and integrate sequences of interactions. We applied our method to more than 40 experiences of a sewing task. The results demonstrate good potential for modeling and obtaining important properties of the task.

Heat map is an important tool for eye tracking data analysis and visualization. It is very intuitive to express the area watched by observer, but ignores saccade information that expresses gaze shift. Based on conventional heat map generation method, this paper presents a novel heat map generation method for eye tracking data. The proposed method introduces a mixed data structure of fixation points and saccades, and considers heat map deformation for saccade type data. The proposed method has advantages on indicating gaze transition direction while visualizing gaze region.

We present a human point of gaze estimation system using corneal surface reflection and omnidirectional image taken by spherical panorama cameras, which becomes popular recent years. Our system enables to find where a user is looking at only from an eye image in a 360° surrounding scene image, thus, does not need gaze mapping from partial scene images to a whole scene image that are necessary in conventional eye gaze tracking system. We first generate multiple perspective scene images from an omnidirectional (equirectangular) image and perform registration between the corneal reflection and perspective images using a corneal reflection-scene image registration technique. We then compute the point of gaze using a corneal imaging technique leveraged by a 3D eye model, and project the point to an omnidirectional image. The 3D eye pose is estimate by using the particle-filter-based tracking algorithm. In experiments, we evaluated the accuracy of the 3D eye pose estimation, robustness of registration and accuracy of PoG estimations using two indoor and five outdoor scenes, and found that gaze mapping error was 5.546 [deg] on average.

Previous research has shown that transcripts generated by automatic speech recognition (ASR) technologies can improve the listening comprehension of non-native speakers (NNSs). However, we still lack a detailed understanding of how ASR transcripts affect listening comprehension of NNSs. To explore this issue, we conducted two studies. The first study examined how the current presentation of ASR transcripts impacted NNSs' listening comprehension. 20 NNSs engaged in two listening tasks, each in different conditions: C1) audio only and C2) audio+ASR transcripts. The participants pressed a button whenever they encountered a comprehension problem, and explained each problem in the subsequent interviews. From our data analysis, we found that NNSs adopted different strategies when using the ASR transcripts; some followed the transcripts throughout the listening; some only checked them when necessary. NNSs also appeared to face difficulties following imperfect and slightly delayed transcripts while listening to speech - many reported difficulties concentrating on listening/reading or shifting between the two. The second study explored how different display methods of ASR transcripts affected NNSs' listening experiences. We focused on two display methods: 1) accuracy-oriented display which shows transcripts only after the completion of speech input analysis, and 2) speed-oriented display which shows the interim analysis results of speech input. We conducted a laboratory experiment with 22 NNSs who engaged in two listening tasks with ASR transcripts presented via the two display methods. We found that the more the NNSs paid attention to listening to the audio, the more they tended to prefer the speed-oriented transcripts, and vice versa. Mismatched transcripts were found to have negative effects on NNSs' listening comprehension. Our findings have implications for improving the presentation methods of ASR transcripts to more effectively support NNSs.

We develop a system for comprehensively evaluating the gaze motions of a person operating a small electronic device such as a PDA or tablet computer. When people operate small electronic devices, they hold the device in their hand and gaze at it. Their hand movements while holding the device are considered part of the movement involved in operating the device. Our measurement system uses a video camera image taken from behind the subject as a substitute for the view camera of an eye-tracking recorder. With our new system, it is also possible to measure the subject's gaze superimposed on the view image by directly inputting the display screen from a small electronic terminal or other display. We converted the subjects' head and hand movements into eye movements and we calculated the gaze from these values; we transformed the gaze coordinates into view image coordinates and superimposed each gaze on the view image. We examined this hand movement in relation to gaze movement by simultaneously measuring the gaze movement and hand movement. We evaluated the accuracy of the new system by conducting several experiments. We first performed an experiment testing gaze movement as the summation of head and eye movements, and then we performed an experiment to test the system's accuracy for measuring hand movements. From the result of experiments, less than approx. 6.1° accuracy was acquired in the horizontal 120° range and the perpendicular 90° range, and we found that the hand motions converted into the angle equivalent to gaze movement could be detected with approx. 1.2° accuracy for 5° and 10° hand movements. When the subjects' hand moved forward, the results were changed into the angle equivalent to gaze movement by converting the distance between the terminal and the subjects' eyes.

The aim of this study is to present electrooculogram signals that can be used for human computer interface efficiently. Establishing an efficient alternative channel for communication without overt speech and hand movements is important to increase the quality of life for patients suffering from Amyotrophic Lateral Sclerosis or other illnesses that prevent correct limb and facial muscular responses. In this paper, we introduce the gaze estimation system of electrooculogram signals. Using this system, the electrooculogram signals can be recorded when the patients focused on each direct. All these recorded signals could be analyzed using math-method and the mathematical model will be set up. Gaze estimation can be recognized using electrooculogram signals follow these models.

We have proposed a novel geometric model of the eye in order to avoid the problems faced while using the conventional spherical model of the cornea for three dimensional (3D) model-based gaze estimation. The proposed model models the eye, including the boundary region of the cornea, as a general surface of revolution about the optical axis of the eye. Furthermore, a method for calculating the point of gaze (POG) on the basis of our model has been proposed. A prototype system for estimating the POG was developed using this method. The average root mean square errors (RMSEs) of the proposed method were experimentally found to be smaller than those of the gaze estimation method that is based on a spherical model of the cornea.

This paper presents a user-calibration-free method for estimating the point of gaze (POG). This method provides a fast and stable solution for realizing user-calibration-free gaze estimation more accurately than the conventional method that uses the optical axis of the eye as an approximation of the visual axis of the eye. The optical axis of the eye can be estimated by using two cameras and two light sources. This estimation is carried out by using a spherical model of the cornea. The point of intersection of the optical axis of the eye with the object that the user gazes at is termed POA. On the basis of an assumption that the visual axes of both eyes intersect on the object, the POG is approximately estimated using the binocular 3D eye model as the midpoint of the line joining the POAs of both eyes. Based on this method, we have developed a prototype system that comprises a 19″ display with two pairs of stereo cameras. We evaluated the system experimentally with 20 subjects who were at a distance of 600 mm from the display. The root-mean-square error (RMSE) of measurement of POG in the display screen coordinate system is 1.58.

We developed a real-world oriented mobile constellation learning environment. Learners point at a target constellation by gazing through a cylinder with a gyro-sensor under the real starry sky. The system can display information related to the constellation. The system has original exercise functions which are not supported by existing systems or products by other research group or companies. Through experimentation, we evaluated the learning environment to assess its learning effects.

People are being inundated under enormous volumes of information and they often dither about making the right choices from these. Interactive user support by information service system such as concierge services will effectively assist such people. However, human-machine interaction still lacks naturalness and thoughtfulness despite the widespread utilization of intelligent systems. The system needs to estimate user's interest to improve the interaction and support the choices. We propose a novel approach to estimating the interest, which is based on the relationship between the dynamics of user's eye movements, i.e., the endogenous control mode of saccades, and machine's proactive presentations of visual contents. Under a specially-designed presentation phase to make the user express the endogenous saccades, we analyzed the timing structures between the saccades and the presentation events. We defined resistance as a novel time-delay feature representing the duration a user's gaze remains fixed on the previously presented content regardless of the next event. In experimental results obtained from 10 subjects, we confirmed that resistance is a good indicator for estimating the interest of most subjects (75% success in 28 experiments on 7 subjects). This demonstrated a higher accuracy than conventional estimates of interest based on gaze duration or frequency.

An eyegaze interface is one of the key technologies as an input device in the ubiquitous-computing society. In particular, an eyegaze communication system is very important and useful for severely handicapped users such as quadriplegic patients. Most of the conventional eyegaze tracking algorithms require specific light sources, equipment and devices. In this study, a simple eyegaze detection algorithm is proposed using a single monocular video camera. The proposed algorithm works under the condition of fixed head pose, but slight movement of the face is accepted. In our system, we assume that all users have the same eyeball size based on physiological eyeball models. However, we succeed to calibrate the physiologic movement of the eyeball center depending on the gazing direction by approximating it as a change in the eyeball radius. In the gaze detection stage, the iris is extracted from a captured face frame by using the Hough transform. Then, the eyegaze angle is derived by calculating the Euclidean distance of the iris centers between the extracted frame and a reference frame captured in the calibration process. We apply our system to an eyegaze communication interface, and verified the performance through key typing experiments with a visual keyboard on display.

In this paper, we propose a flexible gaze detection method using single PTZ (Pan-Tilt-Zoom) camera. In this method, a PTZ camera traces user's face and measures position of their viewing point, so they do not need to fix their head in front of camera. Furthermore, to realize accurate gaze detection, we employ elliptical iris template matching. To reduce calculation amount of iris template matching, we get rough gaze direction by simple method on ahead to decide ellipse shape. In this paper, we also adapt to variation of facial orientations, which will affect to detect viewing position and gaze direction. After several experiments, we examine accuracy of gaze detection and head tracking ability of this system.

Eye contact is an effective means of controlling human communication, such as in starting communication. It seems that we can make eye contact if we simply look at each other. However, this alone does not establish eye contact. Both parties also need to be aware of being watched by the other. We propose a method of bidirectional eye contact satisfying these conditions for human-robot communication. When a human wants to start communication with a robot, he/she watches the robot. If it finds a human looking at it, the robot turns to him/her, changing its facial expressions to let him/her know its awareness of his/her gaze. When the robot wants to initiate communication with a particular person, it moves its body and face toward him/her and changes its facial expressions to make the person notice its gaze. We show several experimental results to prove the effectiveness of this method. Moreover, we present a robot that can recognize hand gestures after making eye contact with the human to show the usefulness of eye contact as a means of controlling communication.