HUMAN-COMPUTER INTERACTION GETS A HELPING HAND.

Computers are one step closer to "understanding" people, thanks to progress in human-computer interaction research at Rutgers University funded by the National Science Foundation (NSF) Directorate for Computer and Information Science and Engineering.

Keyboard and mouse inputs suffice for many users and PC applications. But NSF-funded researchers in a project called STIMULATE are developing systems that mimic other forms of communication that humans use to interact with each other, including eye contact, gestures, touch and voice. The experimental hardware and software may find uses in medicine, the military, disability resources and other fields. NSF funding is about $780,000 for three years.

Computer scientists and electrical engineers at Rutgers have designed Multimodal Input Manager (MIM) hardware that simultaneously receives speech, gaze and tactile signals. Then special software called Fusion Agent assimilates the complex inputs so the computer may respond to subtle signals that humans routinely use to communicate with one another.

A pneumatic "force-feedback" glove, patented by Rutgers, weighs less than three ounces and reads gestures by detecting fingertip positions relative to the palm. It lets the user point at the computer screen, overriding signals from a gaze-tracking camera.

Whereas other gaze trackers require cumbersome headpieces, the MIM's gimbal-mounted unit sits on the desktop and rotates to detect where the user is looking. After a 10-second initial calibration of the infrared detectors, the use can direct a cursor just by looking at a section of the computer screen.

"While we don't foresee that the keyboard and mouse will become obsolete anytime soon," says STIMULATE project leader James Flanagan, "MIM technology opens possibilities for improving current computer applications and for developing entirely new ones that require more-refined modes of human-computer interaction."

The software even detects lip movement to steer a microphone array for use in high-noise environments. For groups of users, the array can home in on the vocal source, even if the person speaking moves around the room.

MIM users at multiple locations can simultaneously interact with each other in a unified, 3D-work environment. Using the Java programming language, the project also produced new cWorld (for Collaborative World) software that lets teams of users construct those virtual environments.