Human Interface Engineering Laboratory, Osaka University


Virtual Tools
Virtual Hand

[Major publications related to this topic]


Tools are expected to provide users with good metaphors in future interactive systems for intuitively manipulating information or virtual objects. Moreover, by using a "virtualized" tool, we can investigate varieties of new tools by simply changing the software parameters of the tool with a standardized interface. Through these experiments, we expect to obtain new knowledge to design a good/comfortable user interface. In addition, we also expect to analyze processes that occur while humans learn to use new tools. We adopt chopsticks for one of the trials of our technique because although they are very simple in form, they do have multiple functions.

Implementation of Virtual Chopsticks

A conventional glove-type hand gesture input device is used to measure the motion of the user’s hand and fingers while handling the virtual chopsticks. But if a one-to-one mapping between the corresponding joint angles is used, the output motion is not always the same as the input motion because of measurement errors, differences of skeletal structures, and so on. The problem is even more serious when the motion data is used in an interactive environment such as a virtual reality system. The motion that a user imagines (mental model) is deviated from the reproduced motion (system behavior), and this seriously degrades the intuitiveness, making the user interface hard to use.

Therefore, we propose a technique to use a multiple regression formula that establishes the relationship between multiple motion data sequences measured by joint angle sensors of a hand gesture input device and the status of the hand gesture while operating the chopsticks. Multiple regression analysis is a method of representing the relationship between the predictor variables and the criterion variable. Here, the joint angles measured by a hand gesture input device are used as the predictor variables, and the distance between the tips of the chopsticks is used to estimate the status of the task of using chopsticks.

Once the multiple regression formula is established for each user, this method should enable even users who cannot handle real chopsticks properly to operate the virtual chopsticks in a way that matches his/her mental image obtained from multiple joint angles, which are measured as a finger motion. Therefore, this formula is used to represent the basic chopsticks. With this method, exact calibration is not necessary, even for a group of users who have a variety of skeletal structures.

Generating sign language by using the user’s intuitive hand action might be a promising application. Application of the proposed method using multiple regression analysis is not limited to operatingl chopsticks. It can also be applied to the motion of other hand gestures and/or any body part that can be represented by a few parameters.

Analysis of Process Occurring while Humans Learn to Use New Tools

The process while humans learn to use new tools can be analyzed by using the virtual chopsticks. Here, it is necessary to use a framework in which the degree of completeness of the user’s mental model can be easily controlled while the fundamentally invariant interface remains. For this purpose, virtual chopsticks can be used as a basic tool, and new tools having several levels of difficulty are developed based on the basic chopsticks. Although chopsticks are one of the most commonly used tools, handling chopsticks is sometimes considered to be a complicated and difficult task to do properly. Based on the basic chopsticks, three different types of new tools are generated by changing the coefficients in multiple regression formula, each with its own level of difficulty, using a standardized interface based on a glove-type hand gesture input device.

basic chopsticks reverse chopsticks

The process of humans learning to use new tools is analyzed with two different approaches, i.e., a performance evaluation that directly observes the human during interactions and an indirect observation of human brain activity by fMRI. By comparing the results, we found that the observation of human brain activity by fMRI is a useful method of evaluating the human interface systems. Although there are many uncontrolled factors and differences between experiments in an ordinary laboratory and in fMRI system, examination of these differences and, if possible, comparison of their effect on brain activities, remain interesting problems for both brain science and human-computer interaction.

velocity chopsticks acceleration chopsticks

Force Feedback for Virtual Chopsticks

We also propose a model of representing a force feedback in manipulating virtual objects with virtual chopsticks. The model is established by using a leverage based on the correct chopsticks handling manner and the force is applied mainly to the index and middle finger. 

Our virtual chopsticks project appeared on the front page of Mainichi newspaper 
(September 28, 1999). --- The picture and text are clickable.

Visual Simulation of Soft Objects


We also propose a system of virtual chopsticks that enables a user to manipulate virtual soft objects intuitively. In this system, the object is defined by a linear FEM (finite element method) model.

[Major publications related to this topic]

Other research projects in Osaka University Human Interface Engineering Lab.