Looking-glass world: learning to assemble the machinery of illusion.

The glare of a projection video screen spills into a darkened room, catching two strangely accoutered figures in its harsh light. Wearing wired baseball caps, tethered, skintight gloves and stereo glasses, the figures bob and weave in front of the screen. Their actions and gestures seem disconnected from the world. Their encumbered heads trace out cryptic patterns; their gloved hands grasp at nothing but air.

The focus of this bizarre pas de deux, however, is not the empty space in front of the screen but the brightly colored, miniature world displayed on the screen. Images of disembodied hands swat tumbling, flexible geometric shapes -- "rubber rocks" -- that deform and sometimes shatter and shriek as they collide with each other and with the tiled walls and floor of a graphically rendered room.

Each change in head position registers on the screen as a change in viewpoint. Each movement of a gloved hand produces a corresponding shift in that hand's screen image, and the rubber rocks respond appropriately.

This simple, interactive video game, set up in a modest suite of rooms at the IBM Thomas J. Watson Research Center in Hawthorne, N.Y., serves as a crowd-pleasing, though rudimentary, demonstration of a technology aimed at providing more natural means of working with information than the keypads, conventional displays and other paraphernalia presently associated with the computer age. It represents one facet of a broad range of research activities that fall under the rubric of an ill-defined, nascent field of study commonly called virtual reality -- though some researchers prefer to use such terms as immersive simulation, artificial reality, telepresence, virtual world or virtual environment to convey the particular flavor of their work.

The IBM project, along with pioneering research at a number of academic and industrial laboratories, demonstrates both the tremendous promise of virtual reality and the great effort required to make it work. Looking to the future, researchers readily envision creating computer-based environments that would allow surgeons to practice new techniques on simulated patients, business analysts to wander through, manipulate and search for patterns in financial data, and armchair tourists to explore and experience exotic locales without leaving home.

At the same time, these pioneer creators of computer-mediated virtual worlds face a host of obstacles, ranging from the primitive state of the technology needed for generating stereoscopic images in head-mounted displays to the serious gaps in knowledge about human perception. The IBM setup alone, for example, requires at least half a dozen high-powered computers and associated equipment -- connected by a maze of cables and controlled by sophisticated, complex software -- to keep its rubber rocks in action.

Nonetheless, virtual-reality technology holds such strong appeal that researchers in a number of laboratories worldwide have mounted serious efforts to overcome these obstacles and bring virtual reality into the real world. In the last two years, these efforts have made virtual reality the subject of extended exchanges via electronic mail among researchers and enthusiasts. It has also served as the focus of a number of conferences and even a congressional hearing.

Despite this attention, however, virtual reality remains an infant technology caught perhaps prematurely in the limelight. "We're at the dawn of virtual-reality technology. It's a technology that has not yet found its focus," says Michael Heim, a philosophy lecturer at California State University, Long Beach, and a computer-industry consultant. He made his remarks in October at a virtual-reality conference in Alexandria, Va., sponsored by the Education Foundation of the Data Processing Management Association.

In one sense, virtual reality is already familiar to readers of skillfully written novels. Successful authors use words to depict such vivid, compelling characters and settings that it's easy to lose oneself in these purely imagined worlds. Hit video games play on the same mind-teasing elements through the creation of interactive environments that plunge players into the middle of the action.

Much of the technology available to virtual-reality pioneers originated in efforts to use computers to create realistic simulations. For instance, advanced computer-based flight simulators now play key roles in the training of both civilian and military pilots. Tank simulators are realistic enough to cause severe motion sickness in novice drivers who push their ersatz vehicles into gravity-defying, stomach-churning maneuvers.

This technology stands at the center of SIMNET, the world's largest immersive simulation. Personnel at military bases scattered throughout the United States and Europe jointly participate in maneuvers on the same simulated battlefield and wage war games on a massive scale entirely within computer-controlled and networked tank, helicopter and fighter-bomber simulators.

Driven by the visions of filmmakers and industrial designers, computer programmers have also developed increasingly sophisticated techniques for rendering realistic, three-dimensional graphic images. Though computer-intensive, these expressive graphic techniques for visualizing information capture hitherto unseen patterns, whether in masses of financial data or the topography of Mars.

Virtual reality represents the convergence of computer simulation and visualization into a single, coherent entity. It also encompasses an attempt to eliminate the traditional separation between user and machine and to provide a means of naturally and intuitively interacting with information.

"We've had to do things in ways that the machine could understand -- using punch cards, keyboards and the like. With virtual reality, you wouldn't even know you were interacting with a computer," says Thomas P. Caudell of Boeing Computer Services in Seattle.

Researchers have already used various combinations of hardware and software to create complex, interactive computer models of human systems, which represent first attempts at creating virtual reality.

At the University of North Carolina in Chapel Hill, a chemist, wearing bulky goggles to see a three-dimensional image on a computer screen and using a special joystick, can maneuver a small molecule so that it docks in the cranny of a gigantic protein -- while literally feeling the force exerted on the incoming molecule by the protein.

On the same campus, an architect, walking on a treadmill while steering a pair of bicycle handlebars, can stroll through a visualization of his or her creation -- in effect, experiencing blueprints brought to life. In Japan, a customer in an appliance showroom can design a new kitchen on the computer screen, then don goggles and glove to wander through the imaginary custom kitchen's array of gleaming gadgets.

At the NASA Ames Research Center in Mountain View, Calif., a researcher, wearing an instrumented glove and peering through a boom-mounted display device resembling a scuba diver's mask, can explore the aerodynamic intricacies of air streaming past an airplane in a virtual wind tunnel (SN: 6/22/91, p.398). In another Ames laboratory, a pilot can skim over a Martian landscape, painstakingly reconstructed from photographs obtained in the 1970s during various missions to Mars.

Each of these well-publicized demonstrations results from a lengthy research-and-development effort. All of them suffer from glitches, and once the initial wonder that it can be done subsides, all display various shortcomings. Most require complex, costly technology that is not always as reliable as its users or creators would wish.

But they attract attention -- so much so that researchers who created these virtual worlds complain that the time they spend shepherding visitors through the displays severely cuts into their ongoing research.

IBM's modest virtual-reality effort, led by Daniel T. Ling, is an attempt to build a generic system for generating virtual worlds. The handful of researchers involved in the project have focused on designing computer programs that afford sufficient flexibility to allow users to readily change the way they interact with and build a computer-mediated virtual environment.

"Virtual reality requires not only unusual devices and three-dimensional graphics but also a lot of complex software," says J. Bryan Lewis, a member of the IBM "veridical user environment" group.

That complexity stems from the need to manage a variety of demands created by linking together many different capabilities. For example, the rubber rocks demonstration includes separate devices and computer programs for generating three-dimensional, stereoscopic graphic images, synthesizing sounds, recognizing speech, simulating the physical behavior of rubber rocks and monitoring head and hand movements.

Weaving together these diverse capabilities produces a whimsical world in which players can verbally order new rubber rocks to appear on the screen. The participants can then grab, hit and shoot these simulated objects, while listening to the synthesized sounds of bouncing, jiggling, colliding and disintegrating rocks.

Simulations play a particularly important role. In Many other virtual worlds, a user interacts with information supplied in a database or with animated graphics that have been prepared beforehand. But in the real-time realm of the rubber rocks, movements and interactions are calculated moment by moment from basic physical principles.

"We feel that the really interesting virtual worlds will include one or more simulations," Lewis says. "We need some life, some surprise, in these worlds."

The IBM group has developed a special rule-based computer program called a "dialog manager" to tie together the disparate elements that work behind the scenes to make a virtual world come alive. By separating what happens within a virtual world and the specific devices and programs used to create and interact with that world, researchers say they can change the number of participants or the world itself with relative ease.

For example, the rules for using hand gestures to bat rocks could just as easily apply to the manipulation of atomic groups within computer-generated images of complicated molecules. When developed further, such a flexible approach might ultimately provide scientists with a wide range of virtual laboratories, in which they could explore molecules, storms or galaxies.

But that's still a long way off.

In the rubber rocks demonstration and in most other virtual worlds, the motion of simulated objects appears somewhat flat and jerky. Users often notice disconcerting delays between a head or hand movement and the registration of that movement on the screen.

With a few notable exceptions, users must also contend with wearing bulky, restrictive head-mounted displays in order to interact with a virtual world. These mask-like displays, which consist essentially of a pair of tiny television sets, produce images so grainy that a user relying only on what he or she can see on the miniature screens would be considered legally blind.

Similar problems afflict the tight-fitting gloves threated with webs of optical fibers needed to sense hand movements. Though commonly used, these gloves respond better to sweeping movements than to subtle gestures, and they remain fragile.

Moreover, generating realistic computer graphics and simulations requires immense computing power that often must be spread over several computers. At the same time, writing and testing the software necessary for creating virtual worlds have so far proved both difficult and error-prone.

The IBM effort is still at an early, experimental stage. "We're building for three to four years from now," Lewis says.

IBM executives say they have no immediate plans for any products related to virtual reality. For now, the company has focused its efforts on developing new ways of visualizing scientific and financial data on computers.

As an instrument of scientific discovery, "the computer is more versatile than the telescope or the microscope, but what has been missing is a good eyepiece," says IBM's C.N. Liu. That's where visualization and, ultimately, virtual reality enter.

The Boeing Co. in Seattle, through its Boeing Computer Services subsidiary, has embarked on a course considerably more ambitious than that of most companies dabbling in virtual reality. Boeing executives now see virtual-reality technology as offering potential solutions to a range of problems encountered in the aircraft industry.

"We need advanced technology to change the way we do design, manufacturing, training and marketing," says Boeing scientist Chris Esposito. "We see virtual reality as a fairly obvious next step."

Boeing already uses computers extensively in the design and production of airplanes. Its new 777 aircraft, scheduled for delivery in May 1995, will be the first airliner designed and engineered entirely within a computer. The 777 program has no need for drawing boards or even blueprints of the traditional type.

By adding virtual-reality capabilities to computer-aided design, company researchers hope to avoid the kinds of maintenance problems that presently be-devil airline employees when, for instance, only a person of a certain height or arm length can tighten a particular nut or reach a defective part. Using virtual-reality technology, designers could readily find out -- while an airplane is still in the design stage -- whether a maintenance worker is likely to scrape a knuckle when replacing a part.

And there are many other situations in which virtual-reality technology may play a role, Esposito says.

As a first step, he and his co-workers developed a remarkably realistic flight simulator for Boeing's experimental VS-X tiltrotor aircraft, so that engineers could evaluate the effect of design changes on its performance. Displayed last summer in Las Vegas at the Association for Computing Machinery's SIGGRAPH '91 conference, the demonstration proved a crowd-pleaser, not just at the meeting but earlier within Boeing itself.

"It caught the attention of a lot of people in the company," Esposito says.

However, because virtual-reality technology is so new and because many questions remain concerning how to translate the massive amounts of data generated during the design of an airplane into forms suitable for a virtual environment, Boeing's team is moving slowly. "This process has to be very much a learning experience," Esposito says.

At Boeing, researchers anticipate that the first products to emerge will fall more in the category of "augmented" reality than full virtual reality. They envision providing factory workers with special goggles on which computer-generated images can be projected. Those images, superimposed on whatever the worker happens to be looking at, would provide specific instructions about how to assemble a particular component, where to position a certain part or what to do next -- whether in the form of schematic diagrams, oscilloscope readings or drill-hole locations.

Because all planes require a lot of custom work, "the idea is to bring information down to the manufacturing floor in a very natural, intuitive way," Caudell says. "This is something we're working on now, and we hope to have prototypes next year in the factory."

In contrast to the somewhat fragmented approach taken in the United States, Japanese corporations have targeted virtual reality as a key technology of the future. Many companies and researchers in Japan see virtual reality as a way to meet human needs by creating intelligent, people-centered products, from computers to automobiles, with applications in business, education and entertainment.

Michitaka Hirose, a leading proponent of virtual reality in Japan, and his collaborators have built up a sophisticated laboratory at the University of Tokyo to experiment with and extend existing techniques. Projects range from the development of advanced head-mounted display systems to novel methods for visualizing complex software that would enable computer programmers to create and analyze programs by manipulating three-dimensional graphic images.

Other researchers have taken a more playfull approach. Koichi Murakami and his colleagues at the Fujitsu Human Interface Laboratory in Kawasaki are studying interactions with a virtual world of friendly creatures floating in a surrealistic fishbowl, where mushroom-shaped forms and monstrous fish respond to hand signals and gestures.

Such an environment serves as a laboratory for studying how humans could interact with intelligent computers to find answers to ill-posed questions. People often don't know exactly what they want to find out, Murakami says. They can use the way a computer reacts to their initial queries to refine a question to get the answer they really need.

"The important thing is to offer . . . an environment where humans can act and objects can react," he says. Virtual reality might provide such a setting.

Programming the four computers required to realize this fishbowl world of responsive creatures represents the biggest challenge, Fujitsu researchers say. To create a sufficiently vivid and compelling world, they must consider a host of issues ranging from the physical characteristics of the simulated pets to the way these creatures should react to various human actions, which can be quite unpredictable at times.

Researchers in industrial, academic and military laboratories don't have the virtual-reality playground all to themselves. Last summer, in a program sponsored by the Pacific Science Center in Seattle, seven groups of children, ranging in age from 10 to 15, spent a week learning about virtual reality and constructing their own virtual worlds.

"We wanted to find out how kids respond to this technology," says Meredith Bricken, director of the education program at the Human Interface Technology (HIT) Laboratory at the University of Washington in Seattle.

In that brief period, each group of eight to 10 children learned enough about the technology and the computer software underlying virtual reality to create their own interactive worlds. After agreeing on a scenario, they used special graphics software to create three-dimensional images of the objects they wanted in their creations. They decided where to put things, which objects were mobile and which ones could be picked up or manipulated in some way. Bricken herself programmed the movements.

"Each group built remarkably interesting, very imaginative and complex worlds," Bricken says. "We had everything from a mountain scene with wonderful trees, farm objects and dirty laundry . . . to a space colony with a monorail . . . in a crater-strewn landscape."

Interestingly, none of these virtual worlds involved conflict; there were no war games. Instead, the youngsters were very protective of their worlds and created interactions with such objects as ghosts, monorails, bears and boats.

Bricken found the results stunning. "They went so far beyond what any of us thought they could do that . . . it was exhilarating," she says.

Despite the hard work and concentrated effort required to complete their projects, the students loved the experience. They also proved perceptive critics, quickly pinpointing many of the inadequacies of present-day practice in building virtual worlds.

"This is an infant technology," Bricken notes. "Yet, "even given the state of the art, the fact that 13-year-olds can in a week build their own worlds is indicative taht it's not all that hard."

In trying to design systems for human use, virtual-reality researchers face many relatively unexplored issues concerning the way people respond physiologically and cognitively to such novel forms of computer-mediated interaction. Investigators need to know much more about how people develop their mental models of the external world and what role human imagination plays in creating illusions compelling enough to fool the mind and eye into accepting the reality of a virtual world.

The construction of virtual environments also raises safety issues, especially when these worlds incorporate force feedback. Right now, a "cybernaut" can step through walls and wander at will through a computer-generated building without fearing the consequences, and a player batting a rubber rock doesn't feel the impact. But that may change with the development of sensors and devices that exert actual forces on a participant. Because virtual environments can allow physically improbable situations -- in some cases, perhaps inadvertently because of a program bug -- the risk of injury or disorientation is very real.

"What we're discovering is that . . . we do not have any tools -- perceptual or design -- for actually building virtual worlds to do things," says Robert Jacobson, associate director of the University of Washington's HIT laboratory. "Eventually we want to get a scientific, more systematic understanding of how these worlds work."

"What we really have so far are demonstratins," says Myron W. Krueger of Artificial Reality in Vernon, Conn. "The technology we are using is quite rudimentary."

Krueger, one of the founding fathers of virtual reality, pioneered a type of interactive medium in which a silhouette image of the user is combined with a computer-generated picture, which the user sees on a large projection screen. Video cameras monitor the person's movements and sent the information to a computer, which responds by shifting or changing the graphic images under its control. Unlike other forms of virtual reality, Krueger's scheme doesn't require the human participant to wear any special gear (SN: 6/22/85, p.396).

With other virtual-reality systems, "you're essentially putting on scuba gear," Krueger contends. "The technology is very intrusive."

But the most critical flaw in present-day systems, he adds, is the perceptible lag between an action and the system's response.

"We all have a long road to travel," IBM's Lewis told an audience at the Virtual Reality '91 conference, held in San Francisco last September. "But let's not get discouraged by the difficulties. Even though we're limited to easy problems at the moment, we can design our software to stay flexible and growable into the future."