Computing art; can a computer be taught to take a painting's measure?

COMPUTING ART Can a computer be taught to take a painting's measure? The "Ocean Park" paintings of contemporary artist Richard Diebenkorn glow with a soft, hazy light. Translucent, luminous colors wash over barely visible skeletons of horizontal, vertical and diagonal lines. Each canvas stands as a window onto an abstract landscape--a serene sea or a stretch of open land.

Hidden within each scene are hints of how the artist created his painting. Underlying lines show through thin layers of paint. One color barely covers another. It becomes possible to trace the order in which the artist put down the elements of his composition.

These subtle clues now form the basis of an attempt to use formal descriptions and computer models to describe what painters do. The idea is to write down a set of rules--a grammar--that would allow someone to analyze the structure of a set of paintings and to generate similar images. Discovering this underlying structure, in turn, reveals something of an artist's mental processes. Such a "design grammar" already exists for architect Frank Lloyd Wright's houses and was used to generate drawings that fooled architects into believing that Wright had done the work himself (SN: 7/7/84, p. 10).

"We tend to think of paintings as unique things, as something uninterpretable," says Curtis L. Carter, professor of aesthetics and philosophy at Marquette University in Milwaukee. "I don't accept that view." Carter also heads the university's Patrick and Beatrice Hagerty Museum of Art.

About a decade ago, Carter wrote several papers on a theory of paintings have language-like features and that a grammar of shapes may be useful for interpreting an artist's work. "When we interpret paintings, we use principles not unlike those that we use in interpreting verbal languages," says Carter.

"Creating a grammar forces you to think clearly and to look clearly," says Joan L. Kirsch, a printmaker and art historian. "Just as one meausres things in science, a grammar provides an objective kind of measurement, an objective way of understanding and communicating our knowledge about a work of art."

Adds Carter, "This gives you a more profound understanding of the structure of a work of art. It gets you beyond the stage of walking mindlessly through a museum and simply gazing in space. It treats art on a more serious level as a fundamental form of human symbolism."

"The grammar serves as a powerful vehicle for expressing insight," Joan Kirsch and her husband, computer scientist Russell A. Kirsch, state in a paper to be published in ENVIRONMENT & PLANNING B: PLANNING AND DESIGN (Vol. 12, No. 4).

For their first try at writing a grammar for painting, the Kirsches chose Diebenkorn's "Ocean Park" canvases. These paintings are roughly geometric in appearance. Made up of lines dividing the canvas into rectangular and triangular areas, the "deep structure" of these pictures is relatively easy to describe in formal terms.

Diebenkorn also leaves traces of the pentimenti, or underlying drawing lines, that define his picture. Most other artists erase or cover up such lines when a painting nears completion. Says Joan Kirsch, "That allowed us to understand his working process a little more easily."

"At the same time," she says, "his work has several levels of ambiguity and informality along with its very formal aspects. That offered a challenge." While the Kirsches couldn't capture the colors, textures or brushwork of a completed painting, they were able to concentrate on the geometric framework on which the artist draped his paint.

"What made it tough," says Russell Kirsch, "is the way [Diebenkorn] builds his abstract paintings. They're very, very subtle. He really relies on the physical properties of paint, on line quality, the nature of color and things of that sort.

"The kinds of tools that we've developed in artificial intelligence and computer science in general," he says, "begin to pale when you start dealing with really difficult problems like describing an important painting." Nevertheless, tools like the theory of algorithms help to illuminate how things are done.

Scrutinizing dozens of Diebenkorn's paintings, the Kirsches soon realized that the artist's line patterns are very tightly constrained. "Every line depends on every other line," says Joan Kirsch. The grammar, therefore, had to include the logical sequence in which the artist put down his lines. Furthermore, it had to model Diebenkorn's fondness for nesting forms with forms.

These observations and many others came together in the form of a grammar consisting of roughly 42 rules. "This doesn't describe in absolutely precise, explicit detail everything that goes on in his pictures," says Kirsch, "but it does represent the kind of choices that Diebenkorn makes."

In the Kirsch scheme, the rules govern where lines can be drawn. The first step may define a narrow vertical or horizontal strip along the edge of an initially empty canvas. Subsequently applied rules decide what kinds of lines fit into these areas and where the lines go. The grammar's rules can also be broadened to specifiy angles, line lengths, region widths and whether lines can be extended outsides of a particular area. In addition, the scheme allows distinctions to be made between differently colored regions, although it doesn't suggest what colors to use.

The formulation of these rules, based on close examination of dozens of Diebenkorn's paintings, required many months. After developing the grammar, the Kirsches analyzed one of Diebenkorn's pictures to see whether the rules, applied in a certain order, could recreate its structure. They succeeded.

The next test of the newly developed grammar involved, in essence, setting the rules free to produce an "original" Diebenkorn. Would the rules, following some randomly defined sequence, generate pictures that anyone could mistake for a Diebenkorn painting?

If the pictures looked genuine, then the rule-makers could claim that they had captured something of the artist's thought processes and commposition methods. The Kirsches produced two test pictures. The procedure was simple enough to do by hand, although a computer program would have done the job more quickly.

When the rule-generated pictures were shown to Diebenkorn, who lives quietly in Santa Monica, Calif., he was surprised. "I looked and felt immediate recognition," says Diebenkorn, " and yet it clearly wasn't may work."

The pictures look like something from a particular phase a few years earlier in his career, he says. "My work has changed a bit since then."

When he first heard of the project, Diebenkorn had been reluctant to cooperate. "I think I had the kind of knee-jerk, artist's reaction to the scientist who tampers with the mysteries of art," says Diebenkorn. "Most artists can be depended on to be ruffled in response to this. I quickly overcame that."

The Kirsches have accomplished a lot, Diebenkorn now says. "It's a very impressive thing."

"IT's very interesting because it seems to reproduce the style of the artist very well," says Marvin Minsky, one of the pioneers in artificial intelligence research at the Massachusetts Institute of Technology. Some people, he adds, are surprised that this short set of rules can do as much as it does.

Russell Kirsch emphasizes that the exercise isn't just an attempt to mass-produce Diebenkorns. "It says not only here's painting that could pass for a Diebenkorn but also here's how it was put together," he insists.

"In much of the 20th century," he says, "artists have been grappling with what they speak of as process." In the splatered painting of Jackson Pollock, for example or in the slashing brushstrokes that make up Willem de Kooning's figures, the way a painting is done seems more important than the actual subject. "The viewer, in a sense, gets swept up in what the artist is doing," says Joan Kirsch.

Because artists often reveal their process of painting in their work, "recognizing that process becomes a way of understanding a work of art," she says. "Identifying 20th century art isn't too hard because most artists go to some lengths to stake out a particular style, a particular way of painting, which is pretty recognizable.

This grammar-building project may also be relevant to computer science and to image processing in particular. Normally, about a million or more bytes of information would be needed to produce a decent representation of a picture by scanning it electronically. But it takes only about 8 bytes of data to describe the rules and steps needed to recreate the basic structure in a typical Diebenkorn painting.

"You get the distinct feeling that if you can do so much with 8 bytes," says Russell Kirsch, "what are you getting with the remaining million or so bytes of information?" Kirsch, now retired from the National Bureau of Standards and working for his own company, the Sturvil Corp. in Clarksburg, Md., was one of the pioneers in developing image processing and techniques for patterns recognition by computers.

"An analysis in which one describes the structure of an image to a machine can carry much of the weight of the image," he says. "If we have computer tools for representing these structural descriptions and manipulating them, then we can do a more complete job of representing the original image."

"[The Kirsches] are doing basic research on a method of analyzing complex visual systems," says Harry Rand, painting and sculpture curator at the National Museum of American Art in Washington, D.C. "That's an exciting frontier."

The context of their work may be somewhat analogous to the evolution of airplanes, suggests Rand. The Wright brothers' flyer, which looks more like a machine for drying laundry than one for flying, now seems very primitive compared with a Boeing 747. Yet the flyer was a necessary first step.

The Diebenkorn grammar is only a first step in analyzing paintings. "We hope our line of development is on the right track," says Joan Kirsch. "We have a lot more to do." This includes work, for instance, on Diebenkorn's use of color and texture--important elements in his paintings.

"These are lots of ways it can be extended," says Minsky. But, he cautions, "there's a fundamental question of how far you can go in reproducing a particular thing that the human mind does without having many of the other mechanisms the mind has. That's a big unknown."

In the field of expert systems, says Minsky, people have been surprised by how far a computer can get even when the machine doesn't know anything about the world in general. "At some point, you have to have more semantic rules," he says, "so that it understands what it is trying to do."

The Kirsches chose Diebenkorn "because he's subject to this kind of analysis," Rand says. "By the time you get to a Rembrandt, it's very different. You have to make different kinds of statements, which may in turn prove virtually imposible to make. But you can't know that until yo get there and perform many of these kinds of experiments."

Russell Kirsch contends that the kind of contributions already made by computer science makes possible a common ground of talk among the artist, the art critic, the art historian and the computer scientist. "It means that the whole business of criticizing and analyzing art can now begin to have some of the benefits of scientific discourse and scientific criticism," he says.

A well-constructed grammar provides a standard notation for examining and criticizing art, says Joan Kirsch. "We all know what we're talking about when we describe something in grammatical terms."

Nevertheless, says Russell Kirsch, "it appears that deep insight and understanding are necesary first. All that we offer are some tools for expressing that insight. The processes for arriving at the insight still remain quite mysterious." It takes a human being to come up with a grammar--or a truly original painting.

"In my work," says Diebenkorn, "I'm continually trying to do it differently. For a picture to come to life for me, it necessitates a series of surprises or maybe one big bang of a surprise. That's the crux of my work. It's surprise that keeps it alive for me."

Says Diebenkorn, "I'm not sure that the computer allows for that."