With the recent advances in IoT, there is a growing interest in multi-surface computing, where a mobile app can cooperatively utilize multiple devices' surfaces. We propose a novel framework that seamlessly augments mobile apps with multi-surface computing capabilities. It enables various apps to employ multiple surfaces with acceptable performance.

We present an interactive modeling system for Japanese castles. We develop an user interface that can generate the fundamental structure of the castle tower consisting of stone walls, turrets, and roofs. By clicking on the screen displaying the 3D space with the mouse, relevant parameters are calculated automatically to generate 3D models of Japanese-style castles. We use characteristic curves that often appear in ancient Japanese architecture for the realistic modeling of the castles. We evaluate the effectiveness of our method by comparing the castle generated by our method with a commercially-available 3D mode of a castle.

This paper introduces a novel skin-stretcher device for gently urging head rotation. The device pulls and/or pushes the skin on the user's neck by using servo motors. The user is induced to rotate his/her head based on the sensation caused by the local stretching of skin. This mechanism informs the user when and how much the head rotation is requested; however it does not force head rotation, i.e., it allows the user to ignore the stimuli and to maintain voluntary movements. We implemented a prototype device and analyzed the performance of the skin stretcher as a human-in-the-loop system. Experimental results define its fundamental characteristics, such as input-output gain, settling time, and other dynamic behaviors. Features are analyzed, for example, input-output gain is stable within the same installation condition, but various between users.

This paper proposes a painterly morphing algorithm for mobile smart devices, where each frame in the morphing sequence looks like an oil-painted picture with brush strokes. It can be presented, for example, during the transition between the main screen and a specific application screen. For this, a novel dissimilarity function and acceleration data structures are developed. The experimental results show that the algorithm produces visually stunning effects at an interactive time.

The purpose of this study is to investigate the effects of device size on non-visual icon search using a touch interface with voice output. We conducted an experiment in which twelve participants searched for the target icons with four different-sized touchscreen devices. We analyzed the search time, search strategies and subjective evaluations. As a result, mobile devices with a screen size of 4.7 inches had the shortest search time and obtained the highest subjective evaluation among the four devices.

There have been many previous studies to facilitate the use of smartphones as remote controllers of PCs. Image-based user interfaces have been suggested to provide fully functioning remote applications. However, most previous image-based interfaces consume high battery power and network bandwidth. Also most users have specific preferences on various applications on remote PCs, but previous smartphone interface systems would not allow users to define their own smartphone interfaces to set their preferences. This paper presents a new smartphone user interface system, SmartUI, for remote PC control. SmartUI is designed as a text-oriented web-based interface, so that it can be used on any smartphone with a built-in web browser while saving battery power and network bandwidth. Moreover, SmartUI enables a user to create buttons on a smartphone; for a quick launch and for shortcut keys, associated with a specific remote PC application. As a result, SmartUI allows a user to create his or her own smartphone interface for remote PC control, while saving battery power and network bandwidth. SmartUI has been tested with various smartphones and the results are also presented in this paper.

As the development of Internet-of-Things is moving towards large scale industry, such as logistic and manifacturing, there is a need for end-users to get involved in the process of creating IoT easily. In this paper, we introduce PaperIO, a paper-based 3D I/O interface, in which a single piece of paper can be sensed and actuated at the same time in three dimensions using the technology of selective inductive power transmission. With this technology, paper material with multiple embedded receivers, can not only selectively receive inductive power to perform paper-computing behavior, but also work as input sensors to communicate with power transmitter wirelessly. This technology allows the creation of paper-based sensor and actuators, and forms an Interent of Embedded Paper-craft. This paper presents the detailed implementation of the system, results of the technical experiments, and a few sample applications of the presented paper-based 3D I/O interface, and finally discusses the future plan of this research.

In this paper, we present a real-time hand pose recognition method to provide an intuitive user interface through hand poses or gestures without a keyboard and a mouse. For this, the areas of right and left hands are segmented from the depth camera image, and noise compensation is performed. Then, the rotation angle and the centroid point of each hand area are calculated. Subsequently, joint points and end points of a finger are detected by expanding a circle at regular intervals from a centroid point of the hand. Lastly, the hand pose is recognized by matching between the current hand information and the hand model of previous frame and the hand model is updated for the next frame. This method enables users to predict the hidden fingers through the hand model information of the previous frame using temporal coherence in consecutive frames. As a result of the experiment on various hand poses with the hidden fingers using both hands, the accuracy showed over 95% and the performance indicated over 32fps. The proposed method can be used as a contactless input interface in presentation, advertisement, education, and game applications.

We present an Interactive Tabu Search (ITS) algorithm to reduce the evaluation load of Interactive Evolutionary Computation (IEC) users. Most previous IEC studies used an evaluation interface that required users to provide evaluation values for all candidate solutions. However, user's burden with such an evaluation interface is large. Therefore, we propose ITS where users choose the favorite candidate solution from the presented candidate solutions. Tabu Search (TS) is recognized as an optimization technique. ITS evaluation is simpler than Interactive Genetic Algorithm (IGA) evaluation, in which users provide evaluation values for all candidate solutions. Therefore, ITS is effective for reducing user evaluation load. We evaluated the performance of our proposed ITS and a Normal IGA (NIGA), which is a conventional 10-stage evaluation, using a numerical simulation with an evaluation agent that imitates human preferences (Kansei). In addition, we implemented an ITS evaluation for a running-shoes-design system and examined its effectiveness through an experiment with real users. The simulation results showed that the evolution performance of ITS is better than that of NIGA. In addition, we conducted an evaluation experiment with 21 subjects in their 20 s to assess the effectiveness of these methods. The results showed that the satisfaction levels for the candidates generated by ITS and NIGA were approximately equal. Moreover, it was easier for test subjects to evaluate candidate solutions with ITS than with NIGA.

Users recently find their interests by checking the contents published or mentioned by their immediate neighbors in social networking services. We propose semantics-based link navigation; links guide the active user to potential neighbors who may provide new interests. Our method first creates a graph that has users as nodes and shared interests as links. Then it divides the graph by link pruning to extract practical numbers, that the active user can navigate, of interest-sharing groups, i.e. communities of interests (COIs). It then attaches a different semantic tag to the link to each representative user, which best reflects the interests of COIs that they are included in, and to the link to each immediate neighbor of the active user. It finally calculates link attractiveness by analyzing the semantic tags on links. The active user can select the link to access by checking the semantic tags and link attractiveness. User interests extracted from large scale actual blog-entries are used to confirm the efficiency of our proposal. Results show that navigation based on link attractiveness and representative users allows the user to find new interests much more accurately than is otherwise possible.

We propose a system that, using video information, segments the mouth region from a face image and then detects the protrusion of the tongue from inside the oral cavity. Initially, under the assumption that the mouth is closed, we detect both mouth corners. We use a set of specifically oriented Gabor filters for enhancing horizontal features corresponding to the shadow existing between the upper and lower lips. After applying the Hough line detector, the extremes of the line that was found are regarded as the mouth corners. Detection rate for mouth corner localization is 85.33%. These points are then input to a mouth appearance model which fits a mouth contour to the image. By segmenting its bounding box we obtain a mouth template. Next, considering the symmetric nature of the mouth, we divide the template into right and left halves. Thus, our system makes use of three templates. We track the mouth in the following frames using normalized correlation for mouth template matching. Changes happening in the mouth region are directly described by the correlation value, i.e., the appearance of the tongue in the surface of the mouth will cause a decrease in the correlation coefficient through time. These coefficients are used for detecting the tongue protrusion. The right and left tongue protrusion positions will be detected by analyzing similarity changes between the right and left half-mouth templates and the currently tracked ones. Detection rates under the default parameters of our system are 90.20% for the tongue protrusion regardless of the position, and 84.78% for the right and left tongue protrusion positions. Our results demonstrate the feasibility of real-time tongue protrusion detection in vision-based systems and motivates further investigating the usage of this new modality in human-computer communication.

XVC (eXtensible Viewer Composition) is an in-vehicle user interface framework for telematics applications. It provides a document-oriented application model, which enables drivers to simultaneously make use of multiple information services, while maintaining satisfactory control of their vehicles. XVC is a new client model that makes use of the beneficial functions of in-vehicle navigation devices. This paper presents the results from usability tests performed on the XVC framework in order to evaluate how the XVC client affects drivers' navigation while using its functions. The evaluations are performed using the Advanced Automotive Simulator System located at KATECH (Korea Automobile Technology Institute). The advantages of the XVC framework are evaluated and compared to a non-XVC framework. The test results show that the XVC framework navigation device significantly reduces the scanning time needed while a driver obtains information from the navigation device.

A component framework for building and operating visual interfaces for context-aware services in ubiquitous computing environments is presented. By using a compound-document technology, it provides physical entities, places, stationary or mobile computing devices, and services with visual components as multimedia representations to enable them to be annotated and controlled them. It can automatically assemble visual components into a visual interface for monitoring and managing context-aware services according to the spatial-containment relationships between their targets in the physical world by using underlying location-sensing systems. End-users can manually deploy and customize context-aware services through user-friendly GUI-based manipulations for editing documents. This paper presents the design for this framework and describes its implementation and practical applications in user/location-aware assistant systems in two museums.

In this letter, we propose a enhanced framework for a Personalized User Interface (PUI). This framework allows users to access and customize virtual objects in virtual environments in the sense of sharing user centric context with virtual objects. The proposed framework is enhanced by integrating a unified context-aware application for virtual environments (vr-UCAM 1.5) into virtual objects in the PUI framework. It allows a virtual object to receive context from both real and virtual environments, to decide responses based on context and if-then rules, and to communicate with other objects individually. To demonstrate the effectiveness of the proposed framework, we applied it to a virtual heritage system. Experimental results show that we enhance the accessibility and the customizability of virtual objects through the PUI. The proposed framework is expected to play an important role in VR applications such as education, entertainment, and storytelling.

In this paper, we propose a framework for virtual reality, I2-NEXT, which enables users to interact with virtual objects by tangible objects in immersive networked virtual environment. The primary goal of this framework is to support rapid development of immersive and interactive virtual reality systems as well as various types of user interfaces. The proposed framework consists of user interface for interactions, immersive virtual environment, and networking interface. In this framework, we adopt several design patterns to guarantee that either developers or users (artists) can easily implement their VR applications without strong knowledge of VR techniques such as programming, libraries etc. One of the key features of this framework is the presence of the device module which supports a natural user interaction in a virtual environment. For example, the proposed framework provides users with tangible objects so that the users are able to manipulate virtual objects by touching real objects. The proposed framework also supports large scale stereoscopic display through clustering technique. To realize the effectiveness of the proposed framework, we have been developing an application for digital heritage reconstruction. Having been through development of the system, we believe that virtual reality technology is one of the promising technologies which enable users to experience realities in a digital space. Detailed explanations of each component and system architecture are presented.

Our group is exploring interactive multi- and hypermedia, especially applied to virtual and mixed reality multimodal groupware systems. We are researching user interfaces to control source→sink transmissions in synchronous groupware (like teleconferences, chatspaces, virtual concerts, etc.). We have developed two interfaces for privacy visualization of narrowcasting (selection) functions in collaborative virtual environments (CVES): for a workstation WIMP (windows/icon/menu/pointer) GUI (graphical user interface), and for networked mobile devices, 2.5- and 3rd-generation mobile phones. The interfaces are integrated with other CVE clients, interoperating with a heterogeneous multimodal groupware suite, including stereographic panoramic browsers and spatial audio backends & speaker arrays. The narrowcasting operations comprise an idiom for selective attention, presence, and privacy-- an infrastructure for rich conferencing capability.

In this paper, we present the Tangible Media Control System (TMCS), which allows users to manipulate media contents through physical objects in an intuitive way. Currently, most people access digital media contents by exploiting GUI. However, it only provides limited manipulation of the contents. The proposed system, instead of a mouse and a keyboard, adopts two types of tangible objects, i.e. a RFID-enabled object and a tracker-embedded object. The TMCS enables users to easily access and control digital media contents through tangible objects. In addition, it supports an interactive media controller which can be used to synthesize media contents according to users' taste. It also offers personalized contents, which suits users' preferences, by exploiting context such as the users' profile and situational information. Accordingly, the TMCS demonstrates that tangible interfaces with context can provide more effective interfaces to satisfy users' demands. Therefore, the proposed system can be applied to various interactive applications such as multimedia education, entertainment, multimedia editor, etc.

Direct manipulation by hand is an intuitive and simple way of positioning objects in an immersive virtual environment. However, this technique is not suitable for making precise adjustments to virtual objects in an immersive environment because it is difficult to hold a hand unsupported in midair and to then release an object at a fixed point. We therefore propose an alternative technique using a virtual 3D gearbox widget that we have designed, which enables users to adjust values precisely. We tested the technique in a usability study along with the use of hand manipulation and a slider. The results showed that the gearbox was the best of the three techniques for precise adjustment of small targets, in terms of both performance data and subject preference.

In this letter we suggest sets of features to classify genres of web documents. Web documents are different from textual documents in that they contain URL and HTML tags within the pages. We introduce the features specific to web documents, which are extracted from URL and HTML tags. Experimental results enable us to evaluate their characteristics and performances. On the basis of the experimental results, we implement a user interface of a web search engine that presents documents grouped by genres.

In 1960, the famous computer pioneer J.C.R. Licklider described a vision for human-computer interaction that he called "man-computer symbiosis. " Licklider predicted the development of computer software that would allow people "to think in interaction with a computer in the same way that you think with a colleague whose competence supplements your own. " More than 40 years later, one rarely encounters any computer application that comes close to capturing Licklider's notion of human-like communication and collaboration. We echo Licklider by arguing that true symbiotic interaction requires at least the following three elements: a complementary and effective division of labor between human and machine; an explicit representation in the computer of the user's abilities, intentions, and beliefs; and the utilization of nonverbal communication modalities. We illustrate this argument with various research prototypes currently under development at Mitsubishi Electric Research Laboratories (USA).