Artificial reality; combining a person's live video image with computer graphics suggests novel ways of working and playing with computers.

ARTIFICIAL REALITY

Combining a person's live video image with computer graphics suggests novel ways of working and playing with computers

Stepping into Myron W. Krueger's creation, VIDEOPLACE, is like entering a wonderland--a whimsical world of playful "critters,' kaleidoscopic colors and sweeping sounds. But VIDEOPLACE is also a serious experiment. It explores and stretches the ways in which people interact with computers. Raise an arm, wiggle a finger, dance, and the computer responds.

"The idea is to create an artificial world in which you can physically participate,' says Krueger, a computer scientist and artist. This is the vision that Krueger, now at the University of Connecticut in Storrs, has been pursuing for more than 16 years.

His goal, he says, "has been to broaden the concept of man-machine interaction' by going beyond the typewriter keyboard, joystick or light pen.

Krueger pictures a "responsive environment' in which a computer observes and interprets human behavior and then responds to these actions with "intelligent' sound and light displays. He looks at this type of environment as a new art form that represents a unique blending of aesthetics and technology. This art form continues to evolve as Krueger and his students tinker with the VIDEOPLACE computer system, adding new features and capabilities.

In the VIDEOPLACE experiment, the silhouette image of the user is combined with a computer-generated picture. The participant sees the result on a large projection screen in a specially equipped room. When the person moves, a computer senses the motion and reacts by shifting or changing the graphic images under its control. In this way, a person's image can appear to make things happen on the screen.

In one VIDEOPLACE program called CRITTER, a coy creature initially flits about on the screen staying just out of reach. If the participant moves toward the creature, it avoids contact. But if the person doesn't move, the emboldened sprite edges closer and finally lands on an outstretched hand. Then the creature beings its arduous climb up the person's silhouette, adjusting to the local terrain as it clambers up. If the person moves during the ascent, the creature clings until the participant slows down. When the sprite eventually reaches the top, it dances a jig in celebration.

But there's more. After its dance, the creature analyzes the situation. If the person's hands are down, it paces nervously. If one of the hands is at shoulder level, it does a flying somersault to land on the hand. If an arm is extended to form a steep slope, the creature dives off the head and rolls down the arm. At the last moment, it catches a finger and dangles. A flick of the wrist dislodges the "critter,' and it heads for the bottom of the screen.

Each time this happens, the creature is programmed to take a somewhat different path, again depending on what the person is doing. When "critter' climbs to the top of the head for the last time, it jumps up and down, causing the person's image to fade away. Says Krueger, "Many participants report an urge to look down at their bodies when their images disappear.'

The success of VIDEOPLACE depends on the ability of computers to respond to movements as they are happening. "This system is designed as a real-time artificial intelligence,' says Krueger.

Over the years, Krueger and a host of students, often building their own equipment, have put together a system that now consists of two general-purpose computers and several specialized processors, including one that executes 40 million instructions per second. Tying it all together is a maze of computer programs that control everything from vision analysis to special graphics.

This system is just the latest scheme in a long line of experiments with "responsive environments.' One of Krueger's earliest efforts involved a pressure-sensitive floor rather than a vision system. As people walked around on the floor, a small symbol moved about on a screen.

"Then I introduced a second symbol,' says Krueger, "and everybody wondered what would happen if they walked their symbol over to the new symbol to get acquainted.' When a person's mark arrived at the spot, the second symbol disappeared only to be replaced by a maze. The whole exercise was merely a sneaky programming ploy to get people into a maze without the necessity of writing a computer program that drew a maze around a person wherever he or she happened to be standing.

Now, with a symbol marking time at the maze's entrance, just about every participant felt the urge to walk the maze. At first, the participants would keep to the paths, taking small steps in the darkened room where this was all taking place. At some point, however, they would realize that because the boundaries existed only on the screen, there was no reason to obey the rules. They'd try to step across a boundary.

"Except that's what I was waiting for,' says Krueger. The first time, the boundary would stretch elastically. A second try would see the person's graphic symbol crumble. In another attempt, the whole maze would shift. "So that whatever they did, I had all these different ways of retaliating,' he says. "It got to be a game between me and them.' And the display, called PSYCHIC SPACE, was a popular feature when it was exhibited in 1971 at the University of Wisconsin's Memorial Union Gallery in Madison.

His ideas, now implemented as VIDEOPLACE but still very much in prototype form, continue to evolve and expand. In CRITTER, for example, about 100 "states' recognized by the computer determine the creature's behavior. Access to a larger computer memory raises that to 1,000 or more choices. "A current goal is to create experiences so complex that they cannot be fully explored in a single session,' writes Krueger in an upcoming issue of LEONARDO.

Krueger has already built equipment that shrinks the participant's image down to the creature's size and provides a three-dimensional scene in which the images can romp. In the future, even the scene will be able to transform itself into fantastic landscapes. Says Krueger, "VIDEOPLACE is an artificial reality in which the laws of cause and effect are composed by the artist.'

In other VIDEOPLACE experiences, participants use their own bodies to control the generation of sounds and colorful patterns, allowing them literally to dance to their own music and to draw to their own motions. In FRACTAL, the idea for which was accidentally triggered by a loose wire, a person's arm movements create complex, animated geometric designs and accompanying sounds. "Rock performers might be able to use it,' says Krueger, "because they want the spectacle as much as the music.'

In all of these interactions, the computer's "reflex' system is now very good at responding immediately to a participant's behavior. But Krueger is interested in developing a "cognitive' system that also monitors and seeks to understand each experience, learns from them and makes strategic decisions. "Such interactions obviously imply a degree of intelligence on the part of the system,' he says, "an almost human awareness of events as they occur.'

Krueger and his work are relatively unknown and are only now beginning to get some attention. His recent presentation in San Francisco at the CHI '85 meeting, sponsored by the Association for Computing Machinery, attracted considerable comment and inquiries from several corporations.

"Ten years ago, no one was interested,' he says. "They just thought I was crazy. But there are actually lots of applications, even though it sounds so frivolous to begin with.'

One obvious application is for sophisticated video games. "You have a way of participating physically in such games,' says Krueger. In principle, a person could climb graphic mountains, swim in graphic seas or even defy gravity and float around the screen.

On a more serious level, children could put themselves in the position of scientists trying to understand the laws of cause and effect on an alien planet. By stepping into a VIDEOPLACE environment, they could explore this "artificial reality,' compare results based on their personal experiences and test their hypotheses. Says Krueger, "They would learn the process of scientific thought rather than memorizing vocabulary and performing mechanical calculations as they often do now.'

As applied to telecommunications, VIDEOPLACE, with video cameras in two locations and a system that combines both of these images with computer graphics, would allow people in different places to share a common experience. A father could play ball with a son thousands of miles away. Working together, two engineers in different offices could resolve a design problem.

Krueger's VIDEODESK idea is, in some ways, the logical extension of video screens that respond to touch. All that would be needed, he says, is a video camera suspended over an empty desk and a flat display screen on a wall. The fingers would do the walking: selecting items from a menu, typing, finger plainting, doing graphic design--any of the myriad things that two hands and ten fingers can specify. And the desk wouldn't be cluttered, as it is now, with telephones, terminals, modems, mice and other gadgets of the computer age.

"You would have quite a comfortable space in which to interact,' says Krueger. "For people who aren't doing word processing, it would probably be sufficient and much more natural because you would expect to have things happen when you reach over to touch them.'

The Mitsubishi Electric  Corp. in Japan is interested in experimenting with these ideas in designs for power system control centers. Researchers at the Microelectronics and Computer Technology Corp. in Austin, Tex., and at Xerox's Palo Alto (Calif.) Research Center, among others, are now considering Krueger's ideas in efforts to each the strain many people feel in trying to extract information from an "unfriendly' computer.

"Anything that can be done with an image is now an ingredient here,' Krueger says. "There are really no limits other than imagination, commitment and resources.'

Krueger himself, while scrounging for supplies and money to continue expanding his system, wants to put his show on the road. He is looking for art galleries, museums and other showplaces where he can display his visions of "artificial reality' and illustrate his contention that computer art is fundamentally interactive.

"VIDEOPLACE is not so much a solution to existing problems,' says Krueger, "as an effort to stretch our thinking about the human-machine interface . . .. As computer interaction becomes the dominant mode of performing work and transacting business, it becomes a significant ingredient in our quality of life. It is time to give the aesthetics of human-machine interaction serious thought.'

Photo: Each frame (above) shows what a participant might see on the VIDEOPLACE screen at various stages during the CRITTER program. The schematic diagram (right) illustrates a typical VIDEOPLACE set-up. The participant, standing against a back-lit screen and facing a video camera, sees the result on a projection screen.

Photo: In one VIDEOPLACE interaction, a person's body movement sculpts a variety of dramatic abstract forms that have a strong three-dimensional quality.