Human Interface Engineering Laboratory, Osaka University
[Major publications related to this topic]
A conventional glove-type hand gesture input device is used to measure
the motion of the user’s hand and fingers while handling the virtual chopsticks.
But if a one-to-one mapping between the corresponding joint angles is used,
the output motion is not always the same as the input motion because of
measurement errors, differences of skeletal structures, and so on. The
problem is even more serious when the motion data is used in an interactive
environment such as a virtual reality system. The motion that a user imagines
(mental model) is deviated from the reproduced motion (system behavior),
and this seriously degrades the intuitiveness, making the user interface
hard to use.
Therefore, we propose a technique to use a multiple regression formula that establishes the relationship between multiple motion data sequences measured by joint angle sensors of a hand gesture input device and the status of the hand gesture while operating the chopsticks. Multiple regression analysis is a method of representing the relationship between the predictor variables and the criterion variable. Here, the joint angles measured by a hand gesture input device are used as the predictor variables, and the distance between the tips of the chopsticks is used to estimate the status of the task of using chopsticks.
Once the multiple regression formula is established for each user, this method should enable even users who cannot handle real chopsticks properly to operate the virtual chopsticks in a way that matches his/her mental image obtained from multiple joint angles, which are measured as a finger motion. Therefore, this formula is used to represent the basic chopsticks. With this method, exact calibration is not necessary, even for a group of users who have a variety of skeletal structures.
Generating sign language by using the user’s intuitive hand action might be a promising application. Application of the proposed method using multiple regression analysis is not limited to operatingl chopsticks. It can also be applied to the motion of other hand gestures and/or any body part that can be represented by a few parameters.
The process while humans learn to use new tools can be analyzed by using the virtual chopsticks. Here, it is necessary to use a framework in which the degree of completeness of the user’s mental model can be easily controlled while the fundamentally invariant interface remains. For this purpose, virtual chopsticks can be used as a basic tool, and new tools having several levels of difficulty are developed based on the basic chopsticks. Although chopsticks are one of the most commonly used tools, handling chopsticks is sometimes considered to be a complicated and difficult task to do properly. Based on the basic chopsticks, three different types of new tools are generated by changing the coefficients in multiple regression formula, each with its own level of difficulty, using a standardized interface based on a glove-type hand gesture input device.
|basic chopsticks||reverse chopsticks|
The process of humans learning to use new tools is analyzed with two different
approaches, i.e., a performance evaluation that directly observes the human
during interactions and an indirect observation of human brain activity
by fMRI. By comparing the results, we found that the observation of human
brain activity by fMRI is a useful method of evaluating the human interface
systems. Although there are many uncontrolled factors and differences between
experiments in an ordinary laboratory and in fMRI system, examination of
these differences and, if possible, comparison of their effect on brain
activities, remain interesting problems for both brain science and human-computer
|velocity chopsticks||acceleration chopsticks|
We also propose a model of representing a force feedback in manipulating virtual objects with virtual chopsticks. The model is established by using a leverage based on the correct chopsticks handling manner and the force is applied mainly to the index and middle finger.
Our virtual chopsticks project appeared on the front page of Mainichi newspaper
We also propose a system of virtual chopsticks that enables a user to manipulate virtual soft objects intuitively. In this system, the object is defined by a linear FEM (finite element method) model.
Other research projects in Osaka University Human Interface Engineering Lab.