Project Ethnography: An Anthropological Approach to Assessing Digital Library Services.

ABSTRACT

OFTEN LIBRARIES TRY TO ASSESS DIGITAL LIBRARY SERVICE for their user populations in comprehensive terms that judge its overall success or failure. This article's key assumption is that the people involved must be understood before services can be assessed, especially if evaluators and developers intend to improve a digital library product. Its argument is simply that anthropology can provide the initial understanding, the intellectual basis, on which informed choices about sample population, survey design, or focus group selection can reasonably be made. As an example, this article analyzes the National Gallery of the Spoken Word (NGSW). It includes brief descriptions of nine NGSW micro-cultures and three pairs of dichotomies within these micro-cultures.

INTRODUCTION

   Questions rained down, and continue to rain down,.... Questions about the
   coherence of life-ways, the degree to which they form connected wholes.
   Questions about their homogeneity, the degree to which everyone in a tribe,
   or even a family (to say nothing of a nation or civilization) shares
   similar beliefs, practices, habits, feelings. Questions about discreteness,
   the possibility of specifying where one culture, say the Hispanic, leaves
   off, and the next, say the Amerindian, begins. (Geertz, 1995, pp. 42-43)

Geertz's words are relevant here. Often libraries try to assess digital library service in comprehensive terms that judge its overall success or failure for their user populations. A variety of methods are used: surveys, usage statistics, standards, and occasionally even focus groups. All of these methods have their virtues and can offer valuable information as part of an assessment process, but misuse is common.

It is tempting to assume coherence and homogeneity among the many populations that use any complex digital library, or at least to assume discrete boundaries between certain populations. Students, for example, from a population that could include people from high school through graduate school and English majors to engineers. A bright high school student may also be taking college classes, and an undergraduate could well have an English-engineering double major. It is also tempting to assume that the developers of a complex digital library project have agreed on common service goals rather than separate (though, it is hoped, symbiotic) priorities. A service goal of making material accessible might, for example, mean a search algorithm to one of the developers, index structure to another, and subject categories to a third. Treating them as indistinguishable parts of a common product loses key information which would help to identify problems and improve services.

This article does not offer anthropology as a substitute methodology for evaluating digital library services. Anthropology as practiced today seeks to be relatively non-judgmental, even in a good cause. It tries instead to observe accurately and to lay out the dynamics of interactions in ways that explain situations and behaviors. This article's key assumption is that the people involved must be understood before services can be assessed. Its argument is simply that anthropology can provide the initial understanding, the intellectual basis, on which informed choices about sample population, survey design, or focus group selection can reasonably be made. It offers a first step, but one which, if ignored, can trip the most sophisticated evaluation scheme.

METHODOLOGY

The standard method for research in cultural anthropology is to find a set of people, learn their language and everything else known about them, and then live with them long enough to come away with new insight and understanding. At one time, the people tended to come from remote tribes, like Margaret Mead's (1932) Samoans. Later the pool grew to include ex-colonial territories--people from complex, but non-western, civilizations, such as Clifford Geertz's (1956) Indonesians. More recently, cultural anthropologists have taken an active interest in aspects of contemporary western society such as John Borneman's (1992) Germans or Bonnie Nardi's (1999) corporate librarians. The methodology for this article follows a similar pattern.

Language is a particularly important aspect of the methodology, even when an English speaker is dealing with other English-speaking Americans. Words do not always convey a simple dictionary meaning, especially across cultural and disciplinary boundaries. "Research," for example, means something with books, articles, archives, and footnotes to the historian, but generally implies mathematics, experimentation, hypothesis, and results to an engineer. The same word conjures up different approaches and different products to differently trained people. The nuances of meaning matter, especially in understanding what the real service goals of a project are. For a study like this, the language training consists not of foreign words but of foreign concepts, acronyms, even symbols (e.g., [Sigma]).

The human subjects in this case are university people, librarians, historians, engineers, education faculty, computer professionals, and others. Since I am a trained historian, a computer professional, and a librarian, I understand the language, the specialized words, the acronyms, and the implicit meanings of three of the subgroups. This is crucial in being able to describe their interests and intentions faithfully. I am also aware of how poorly I understand the meanings of, for example, the engineers, whose mathematical discourse far exceeds my last meager courses in calculus over thirty years ago.

The participant-observer must play two roles simultaneously. It is not always easy. One example of this problem comes from Frank Hamilton Cushing. In 1879, that influential ethnologist went to live among the Zuni and became so completely one of them that he participated in their secret rituals. Ultimately, he became a "Bow Priest," and destroyed many of his notes rather than betray Zuni secrets (Schoumantoff, 1999, pp. 143-44). Where does the participant leave off and the observer begin? There is no simple answer. Geertz (1995) writes: "It is a matter of living out your existence in two stories at once" (p. 94).

Use of the first person in anthropological articles has always been fairly common. It reminds readers of the filter through which they are viewing the world. The references in the body of this article come mainly from my notes, my memory, my records of conversations. I deliberately avoid naming individuals and quoting conversations as I have normally done when writing oral histories because of the ongoing, active, ever-delicate work relationships that could easily be damaged.

The rules of evidence in anthropology lack precision compared to some other social sciences. Clifford Geertz in particular has thought about this issue:

   The ability of anthropologists to get us to take what they say seriously
   has less to do with either a factual look or an air of conceptual elegance
   than it has with their capacity to convince us that what they say is a
   result of their having actually penetrated (or, if you prefer, been
   penetrated by) another form of life; of having, one way or another, truly
   "been there." (Geertz, 1988, pp. 4-5)

      ....Such, such are the facts. Or, anyway so I say. The doubts that
   arise, whether in me or my audience, have only very partially to do with
   the empirical basis on which these accounts, or others like them, rest. The
   canons of anthropological "proof" being what they are (mimicries of sterner
   enterprises like mechanics or physiology) that is, indeed, how such doubts
   are most often phrased and, to the degree they are, most often quieted.
   Footnotes help, verbatim texts help even more, detail impresses, numbers
   normally carry the day. But, in anthropology anyway, they remain somehow
   ancillary: necessary of course, but insufficient, not quite to the point.
   The problem--rightness, warrant; objectivity, truth--lies elsewhere, rather
   less accessible to dexterities of method. (Geertz, 1995, pp. 17-18)

The examples cited in this article are not verifiable except to the limited extent that a few other people heard the same words (though not necessarily the same meanings).

This imprecision may bother some readers, but it lies at the core of the evaluation issue. The most careful survey, the most rigorous statistical test, depends ultimately on the meanings that the words, categories, even the numbers, convey. The worst evaluation disasters occur when the respondent does not understand (or misunderstands) the questions, or when the people responsible for the service being evaluated throw the results into the life-imprisonment of a deep file drawer because the results miss all the issues they could use to make improvements.

ABOUT CULTURES AND MICRO-CULTURES

Culture represents a nexus of shared meaning. It can be used in a broad sense to refer to "western" or "Asian" culture or more narrowly to refer to "German" or "American" culture or still more narrowly to refer to "midwestern" or "Afro-American" culture. The number of possible distinctions has no obvious limit. The culture of a nuclear family can, in fact, differ from its neighbor: different holiday traditions, different vacation preferences, even private words loaded with special meaning (sometimes understood by the spouses alone). In this article, the word micro-culture refers to units of shared meaning as small as professions, departments, and interest groups. The reason for this specialized word, instead of more standard descriptions, is that it evokes the range of anthropological discourse which, for all its flaws and imprecision, offers a theoretical framework for analyzing the social processes involved in service evaluation.

Within the culture of an academic institution, this variety of micro-cultures is easily recognized. The boundaries have some sharp edges, but more that are indistinct. How would one classify a woman who, for example, is working on a library degree (and therefore is a graduate student), but has a doctorate in the history of science (therefore belongs both to the natural sciences and the humanities), and is concurrently teaching as an adjunct at a nearby peer institution (and therefore counts as faculty) ? Such a person belongs to multiple micro-cultures and may react to a particular digital library service from any one of these identities or from another equally important identity altogether.

Most people are, in fact, blends of micro-cultures. One of the important factors in establishing an evaluation system is to understand which cultures and micro-cultures matter. That the "end user" matters most is a common gut reaction, but it may not be true. For a grant-funded project, for example, the real value (in monetary terms) of the service may depend entirely on how the funding agency's project officer perceives the work. That one person can make or break the next year's allocation.

The planned end-user may also differ from the actual end-user. My own office, the Digital Sources Center at Michigan State University, made some public-domain Ku Klux Klan pamphlets available on the Web for classes in American radicalism. Later we found that the Klan itself linked to these materials (P. Berg, personal communication, April 17, 1998). A user-based survey about this service could well involve enough Klan members to give unexpected results. They might, for example, find links to Black Panther pamphlets within the same collection offensive and ask that they be removed. For reasons which have to do with our own cultural values, we would not do so. We would also not knowingly include Klan representatives in a focus group for evaluating that particular digital library service, even if such a person were a student on our campus. Our reaction to members of that particular micro-culture affects our opinion of their responses so greatly that their evaluation of the service becomes irrelevant, and we exclude them intentionally from our definition of the end-user population, even though they actually use the materials.

In the modern workplace, the micro-cultures interact more than they did in traditional hierarchical corporate, or even academic, organizations. In a recent article, Bonnie Nardi and her co-authors argue:

   that it is increasingly common for workers to replace the organizational
   backdrop and predetermined roles of old style corporate working with their
   own personal assemblages of people who come together to collaborate for
   short or long periods. These assemblages are recruited to meet the needs of
   the current particular work project. (Nardi, Whittaker, & Schwarz, 2000)

Most of the second round of Digital Library Initiative grants, and many of the first, involve collaborations that cut across traditional fields--the old style organizational backdrop for corporate academe. Library and computing (or engineering) partnerships are particularly common. Examples can be seen in "Project Prism" at Cornell (http://www.prism.cornell.edu/), "Emulation Options for Digital Preservation" at the University of Michigan and University of Leeds, and the "National Gallery of the Spoken Word" (NGSW) at Michigan State University (www.ngsw.org). Most of the examples in this article will come from the NGSW, whose four-way partnership includes the university library, the College of Engineering, the College of Education, and MATRIX (the "Center for the Humane Arts, Letters, and Social Sciences Online") in the College of Arts and Letters.(1)

What follows is an ethnographic look at the culture of the NGSW and its many micro-cultures. The argument is simply that examining these is essential to understanding the developers' intentions about both services and end-user populations. Special emphasis will be put on what these concepts mean for each of the micro-cultures involved.

THE NGSW WORLD

Origins matter in defining a culture, and the origins of the NGSW lie as shrouded in myth as any ancient cult, even though the project is a scant two years old. At the official kick-off ceremony with an audience of provosts, deans, and visiting dignitaries, one of the co-principal investigators told the story of how a friend approached her at church and proposed a partnership to go after the grant. Another remembered its origin as stemming from a conversation with the vice-provost who recommended, as only a vice-provost can, a partnership with the Computer Science Department, which in its turn made a link to a signal-processing engineer. Yet another co-principal investigator suggested that the true origin dates from his vision to use materials front the Vincent Voice Library (VVL) years before the grant was written.

Friendship, institutional ties, and vision each played a role in the project's origins, and each tends to define the project's nature, purpose, and measures of success in a somewhat different way. The text of the original grant proposal could, and perhaps in theory should, provide a common basis for evaluation, but the text has thus far had little value as a common reference point. The co-principal investigators rarely refer to it or quote from it in their discussions, except occasionally regarding financial matters. An exception occurred during a discussion of applying Bayesian statistics to indexing. One person insisted that something like that had been included in the proposal, but a search of the digital copy of the final draft produced no references to Bayes or Bayesian. The reference probably existed in some version of the text but not the final one.

The proposal text fails to provide a unifying set of principles, in part because no individual could write the proposal as a whole. No person's expertise ranged sufficiently widely to encompass all of the engineering, computing, library, and educational issues. The first draft was a simple composite of uneven texts from each of the co-principal investigators. One saw it as a prototype, a throw-away version to get some of the basic ideas on paper, another reacted in horror at it as amateurish and disorganized. The rewrite blended parts with a heavy hand. One author worked on it remotely by e-mail from Australia, another tried to talk through the ideas before writing them down. The process bogged down so badly that the group went to the vice-provost the Friday before the proposal was due to talk about waiting for the next round. The library director made the key comment: what is there to lose? Even if the text is bad, others might be worse. The submission went forward, but with a strong sense that the project description was seriously flawed.

Those who have worked in a complex grant culture will recognize both the variety of origin myths and the disdain for the defining text as ordinary, even predictable. A genuine partnership, with no single commanding leader, and participants with national standing in each of their disciplines, cannot work out the nuances of vocabulary and priority for a five-year project in the few months of intermittent effort prior to submission. Some modest dissonance may in fact represent the freshness of ideas and vigor of thought that were the true reasons for funding in the first place: a healthy red-cheeked bloom of active intellectual engagement. Unfortunately, this does not help to determine service objectives.

Evaluation is an explicit part of the NGSW proposal, and an external evaluator has a budget line in the subcontracts section (National Gallery of the Spoken Word, 1998). Evaluation received little discussion during the proposal-writing period. It was almost an afterthought, a last-minute addition by those accustomed to the NEH requirements for project evaluation for teaching-related projects. The evaluator is himself a statistician who has strong ties to one (and only one) of the four co-principal investigators and brings substantial experience with educational but not engineering or library settings. He has met with the whole group only once, at the very start, and mainly discussed indexing schemes, not user populations. The general sense is that his work will come mainly at the end, though some murmurings of concern about how to define who and what gets evaluated have surfaced at the edges of meetings, those important periods just before and just after the formal agenda when friend collars friend and seatmate turns to seat-mate. Bonnie Nardi's "intensional" (a combination of "intentional" and "tension") networks have been discussing it, even if the project team as a whole has not (Nardi, Whittaker, & Schwarz, 2000).

NGSW MICRO-CULTURES

A cultural map of the NGSW world is difficult to draw. The first temptation is to accept the institutional boundaries as if they represent the borders between say, France, Germany, Belgium, and Holland. It is too simplistic but cannot entirely be discounted. Such a map would contain the following "lands."

Library

The library's collections are in the top quarter of the Association of Research Libraries in terms of sheer number of volumes. The number of professional staff, however, lies in the second quarter and ranks near the bottom of the fourth quarter in terms of the ratio of professional staff to full-time students (Association of Research Libraries, 2000). Although these figures suggest understaffing, the library has the advantage of an energetic new director whose willingness to put resources into information technology has resulted in the establishment of a Digital Sources Center, plus a growing number of grant proposals and grant-funded projects. The library has significantly increased the number of Ph.D.s on staff in recent years and has recruited vigorously from the best library and information schools in the country. The new recruits are not necessarily young. They bring outside life and work experience and choose to work at Michigan State in part because they perceive it as (and it sells itself as) a place of opportunity. The librarians have faculty status. They are expected to publish. There is a strong sense that the local culture has changed. The three librarians directly involved with the NGSW at present all have doctorates in history, and all have their library, degrees from the same school (University of Michigan), though from somewhat varying eras and specialties.

The library has had two explicit goals for the NGSW. The first is to preserve its large collection of reel-to-reel tapes of speeches, oral histories, and other forms of spoken-word recordings. Since the collection is estimated to have as many as 50,000 hours of sound, this could be no small task. The second explicit goal is to bring those materials under better bibliographic control. Only a small portion of the works have been cataloged in MARC, using an approach that treated each segment as if it were a separate monograph. Another implicit goal is to establish copyright rules and do a systematic check of the collection since the copyright rules for sound are complex and not always well understood. The library's understanding of NGSW service goals focuses mainly on access and preservation. Its traditional user populations include the broadest possible range of students, graduate students, faculty, and potentially all Michigan citizens as part of the university's explicit "land grant philosophy." The library is also concerned with standards setting, both for the preservation and the bibliographic control issues, which makes librarians at other institutions another user population for the NGSW work.

MATRIX

Technically, MATRIX belongs to the College of Arts and Letters. It was created in the last few years to be the local center that housed HNET (Humanities and Social Sciences Online, http://www.b-net.msu.edu/), an independent international organization that runs e-mail lists and review services in the humanities and social sciences. The (elected) executive director of HNET is also the (appointed) executive director of MATRIX (MATRIX: Center for Humane Arts, Letters, and Social Sciences Online, http://www.matrix.msu.edu/). He is an American historian who focuses mainly on political history and has just published a book on American voting patterns. Much of the HNET/MATRIX funding comes from grants. Few of the staff receive permanent university funding, which means that grant-writing and grant-getting are the lifeblood of the organization. MATRIX has only about a dozen full-time employees, some graduate students, and several computer professionals. It handles the accounting for the NGSW, and its director is the official "project director." MATRIX works closely with a political scientist at Northwestern, and with a number of people from the College of Education, though not the NGSW co-principal investigator from there. MATRIX also works closely with the African Studies Center on campus and is becoming increasingly involved in African-related grants. It has excellent national and international contacts.

In a narrow sense, MATRIX's goals for NGSW focus chiefly on education with one explicit focus on undergraduate college teaching and another on high school curriculum. It has worked with the former especially in previous NEH grants. Much of MATRIX's work for NGSW has to do with the online interface, and its staff have looked at providing different interfaces for teachers, high school students, college students, and researchers--an end-user population almost as broad as the library's. In a broader sense, MATRIX also takes an interest in the standards and preservation issues. It works with a number of national organizations on these issues, among them NINCH (National Initiative for a Networked Cultural Heritage, http://www.ninch.org).

Engineering

MSU's College of Engineering routinely brings in significant amounts of outside money. Signal processing, the area concerned with the NGSW, has only one faculty member, a full professor, and a number of graduate students. Mathematical and computing algorithms make up much of the professional discourse. The engineering co-principal investigator works closely with a fellow engineer at the University of Colorado at Boulder. It is this second engineer who will do most of the research on searching the digital sound directly, while the MSU partner will focus more on watermarking. The two men have known each other for years and have worked together before, while at widely separated institutions. The NGSW grant is not as significant for either of them as it is for the other partners, because NSF-administered grants are a standard part of their work life.

The Center for Spoken Language Understanding (CSLU) in Boulder is roughly the size of MATRIX and is also heavily grant-funded (mainly by Defense Department sources). The CSLU engineer working with NGSW is one of the founding members of the CLSU and is its associate director. It is a computing-intensive environment with strong ties to linguistics. Many of the graduate students come from China, India, and elsewhere outside the United States.

The goals for the engineers are more specific than for MATRIX or the library, and they are concerned with not promising more than they can reasonably deliver so that they do not get a reputation for being like the scientists who claimed to have invented "cold fusion" (personal communication, May 30, 2000). CSLU has several keyword/phrase and "gisting" algorithms (i.e., algorithms that give the gist of a speech) that they can modify for searching the VVL sound files. These algorithms use computer-based models to match words or phrases in digital audio streams. The engineers are concerned to include probability estimates of the word-recognition accuracy and are concerned that metadata records of the search results have the flexibility to be updated as search algorithm progresses through generational improvements. The engineers' primary audience is explicitly their peers and their funding agencies rather than any shadowy and distant end-users. Yet they have most consistently raised issues about what end-user questions to expect, perhaps because they make no pretense of knowing who or what that end-user is.

Education

MSU's College of Education is considered to be among the country's best. The faculty member associated with NGSW belongs to the Department of Teacher Education and works largely alone. She has been the assistant superintendent of a big-city school system in the Eastern United States and runs an active consulting practice among Michigan school districts. She is the only female co-principal investigator and the only person of color. Much of her work for NGSW involves close contacts with superintendents and teachers, particularly in disadvantaged school districts. She has little clerical or other administrative support. She spends more time working with people and getting practical results than with grant-writing or purely academic research.

The school systems involved with the NGSW include the tiny rural community of Baldwin, the Detroit suburb of Oak Park, and the city of Benton Harbor (National Gallery of the Spoken Word, 1998). They have a standard bureaucracy of superintendents, principals, teachers, and specialists. Their interest in the NGSW focuses specifically on how the sound files can enhance existing curriculum and educational priorities, which are set, in part, through a statewide system of standardized tests. Many of the students in these systems come from disadvantaged backgrounds and have reading problems. One hope is that they will respond more eagerly to oral than to textual sources. Some work has been done to select particular teachers, and thus classrooms filled with actual nameable students to serve as end-users. Of course the teachers also form an end-user population, as do the principals, superintendents, school boards, even ultimately the district voters.

RELATED MICRO-CULTURES

If the metaphor of the cultural map of the NGSW can be carried further, several other external powers influence the service goals and end-user populations. These entities have their own expectations and ways of doing business and see the NGSW as falling at least partially into their zones of influence.

National Science Foundation (NSF)

In a very immediate financial sense, the NSF is the end-user that matters, because its staff decide each year whether the project is working well enough for funding to continue. NSF provides federal funding for a vast array of natural science, social science, and computing projects. Its headquarters lie in Arlington, Virginia, in a modern fortress-like building with relatively strict security for anyone entering or leaving. Although NSF's mission is ineffably broad, the head of the directorate in charge of the Digital Library Initiatives has a well-articulated vision that includes promoting multi-media research, geo-spatial information systems, and international collaboration (Lesk, 1997). NSF has moved forward aggressively in all of these areas in recent years despite a shortage of staff, especially clerical help.

An important part of the NSF management of the Digital Library Initiative is regular required conferences where the principal investigators and key staff of each of the projects meet and mix. This catalyst for cross-institutional interaction has resulted in some significant changes for the NGSW. At the Ithaca, New York, conference (October 1999), NGSW library staff recognized the value of using Encoded Archival Description (EAD) for doing bibliographic description of the sound files. This led to a major change in service delivery by shifting the contents description from an AACR2-defined MARC-based monographic metaphor which had long been used, but had never fitted the eclectic sound files especially well, to a more flexible XML-based archival collection metaphor (AACR2 is Anglo-American Cataloging Rules, second edition; XML is eXtensible Markup Language, a form of SGML or Standard Generalized Markup Language).

Another explicit, though often understated, NSF expectation is the publication of research from the grants in academically respectable peer-reviewed journals. This is one of the ways in which the scholarly community can measure the effectiveness of NSF grants which helps, in turn, to persuade Congress to support NSF appropriations. Congress could well be seen as the ultimate end-user for NSF and the projects it supports. If Congress cuts funding, the programs die. NSF staff are eager for Congress to hear stories about how the results of funded projects have been used by, or given practical help to, people or activities.

MSU Computing

The MSU computing staff is not large for a university of its size and has suffered some losses lately because of the booming economy. The same vice provost has charge of the library and the computer center. Contacts between the two units are frequent and generally friendly, though fraught with the usual cultural misunderstandings between such different groups. The computing staff manages the infrastructure that NGSW will use for delivering the digital sound, which gives them a strong interest in how the services are delivered. Because of the quantities of audio data involved, they have expressed concerns about the effect on the campus network--whose users are among their primary service clients. MSU is an Internet2 participant (http://www.internet2.edu), and NGSW staff have demonstrated at an Internet2 applications conference in Ann Arbor.

Data storage is another computing staff concern. It remains unclear how much digital sound can (or should) be kept online at any one time. The more sound that is off-line the greater the potential inconvenience for end-users, particularly researchers who want an obscure item. Back-up systems affect service delivery standards too. Tape back-ups of disk arrays will slow systems and require staff-time, storage space, and planning, all of which are costs not included in the grant.

University Administration

At a 43,000 student Carnegie I Research university, even grants as large as $3.6 million do not rate more than occasional attention from the upper reaches of the administrative hierarchy. The vice provost sends a request from time to time for another set of PERT charts to show progress or lack thereof. And the relevant deans occasionally ask one of the co-principal investigators to talk to a group or to contribute a few words to an article in an alumni publication. The interest is real if irregular. The administration's goal for the grant is not unlike its goal for the football team: that it succeed, that it bring credit to the institution, that it make alumni and friends and perhaps state legislators proud of their association with the school. These are the end-users for much of the administration's work, and the more the NGSW service objectives satisfy them, the easier it is to keep the institution funded and functioning.

Corporate Partners

NGSW has only one official corporate partner, a small firm specializing in technology for the visually impaired. Several other companies have approached it, though. These range in size from Dow Jones giants with plants and offices worldwide to high-tech start-ups with a dozen or fewer staff and a single sound-oriented product. No agreement has yet been reached with any. Their interests in the project vary as well. The giant has its own grant program, which could supplement the government money, and it hopes to build a relationship which will ultimately lead to sales. The start-up wishes only to have access to the digital sound so that it can honestly say that its device has purposes other than playing pirated music. For these firms, the end-users are both customers and investors. Their influence on the NGSW at this point is negligible but, when federal funds run short, their resources may grow more tempting and their goals more of an issue.

Standards Groups

Several standards groups, both formal and informal, take an interest in the NGSW. An informal group has met for dinner at the American Library Association for the last several years to discuss the best practices for converting and maintaining sound in digital formats. Originally the word "preservation" was used, but that brought forth strenuous objections from one influential member. The offending word was dropped in order to make progress on the idea. Members of NISO (National Information Standards Organization, http://www.niso.org) committees, and the board have talked with one of the principal investigators about submitting a proposal, or perhaps a proposal for a proposal, on digital sound standards.

Similar conversations have taken place with the Library of Congress. The Association of American Archivists which, with the Library of Congress, sets the standards for the official EAD DTD (the Document Type Definition for Encoded Archival Description), may well also be involved with NGSW-recommended modifications to handle special tags for digital sound. The service objectives for these kinds of organizations have to do with detailed technically efficient proposals that can find broad acceptance among the professional audiences that are their end-users. These standards groups are particularly important for NGSW because, without their endorsement, much of the research and work remains idiosyncratic and local, and it becomes liable to revision when a better idea comes along.

MICRO-CULTURES DICHOTOMIES

Although these official institutional boundaries demark an important set of micro-cultures, several pairs of contending micro-cultures within these institutional groups also influence the outcome. These dichotomies occur both within units and within the minds of individuals. The principal investigators seem particularly liable to shifting sides, depending on how their training and experience matches particular circumstances.

Humanists versus Technicians

The humanists include historians, writing teachers, educators, and linguists. In general, they agree on a broad audience of students and researchers and on service expectations that resemble the search capabilities of a library's online catalog. They contributed most of the words to the original proposal and, perhaps because words are their principal tools, they dominate debate during project meetings. They tend also to misjudge the degree to which they really understand technical issues and often try to state service goals which the technicians shoot down as impossible. One example occurred during a discussion about searching the digital sound files, where several of the humanists realized for the first time that the technicians were proposing something quite different from the equivalent of full-text searching. One of the technicians told me later that he had explained this at least twice earlier in plain terms and wondered why anyone was surprised. What the humanists seem to have misunderstood was the engineering timeline. They imagined the caveats and limits referred only to unimportant short-term steps, not project-length goals.

Local versus National

Because all of the principal investigators are reasonably well-known people in their fields, they participate in a national and, in fact, international research culture, whose interests and demands match imperfectly with local institutional needs. One simple example comes from the library, where the Voice Library staff have a strong interest in ensuring that the whole collection gets put into digital form. They have always thought of the purpose of the grant in those terms, and reminders about the nationally-oriented research and best-practice missions leave them with the unsatisfied look of people not being treated quite fairly.

The engineers feel the problem too. Their reward and promotion system depends strongly on national recognition. For them it is important that their contributions to NGSW also contribute to their basic research mission. They are by no means being uncooperative when they shy away from tasks which meet only local needs. In their world, the balance point between local and national obligations simply occurs in a different place.

Expansion versus Completion

The project contains a mix of expansionists and completionists. The former want to emphasize tasks which will bring in additional money to keep the project growing. The broad promises of the original proposal encourage this since, in some real sense, expansion is the only way to accomplish some semblance of its goals. The completionists want to define, in a clear and measurable way, what subset of work can be done with the money available. It is not that they want the project to stop after five years, but they view a job well-done in terms of well-planned achievable goals and on-time accomplishments.

Both sides talk as if they agree with the other, and both would ideally like to accomplish both aims. The problem lies in the details, since the completionists' plans seem too constrained to attract new money, and the expansionists' initiatives seem at odds with getting the essential work done. One confirmed completionist pushes routinely for more PERT charts. One incorrigible expansionist flies frequently to Washington to prospect for new money.

CONCLUSION

This article has offered brief descriptions of nine micro-cultures and three pairs of dichotomies within these micro-cultures. And yet this was a simplified and abridged list: not an exhaustive survey, but a sample of the complexity that big projects like NGSW involve. It did not even include all of the external partners, such as the Chicago Historical Society (which is contributing sound files, particularly its collection of Studs Terkel tapes) and Northwestern University (which has the "Oyez Oyez Oyez" [http:// oyez.nwu.edu] and "History and Politics Out Loud" [http:// www.hpol.org/] projects).

The problem with communication between these micro-cultures is that they use the same words with different meanings. To the librarian, for example, the word "catalog" has an implicit relationship to MARC and AACR2 that never occurs to the historian, educator, or engineer, who ask in meeting after meeting what those (or other) acronyms stand for in a vain hope that the words representing those initials will convey something approximating a comprehensible explanation. They do not, of course, any more than naming an integral sign clarifies a complex formula to the nonmathematician. This Tower of Babel becomes even worse when discussing broad terms like "service goals" or the definition of "end-user population"--worse but not impossible.

One of the steps to accurate measurement is to try to express the service goals in terms that have the same meaning to both evaluator and respondent. For example, a broad survey question about the "success" of a query that searches a digital sound file is probably meaningless in terms of any constructive feedback to the engineers, but that same question could be useful to a group of educators whose interest lies mainly in the student's perceptions. Similarly, a query about the "usefulness" of a chronological interface may get a low score from an engineering graduate student who cares chiefly about how to improve the word-matching algorithm. And that result might merely perplex the historian who worked hard to craft that particular tool.

The point is that a useful evaluation of digital library services needs to include an understanding of the nuances of the meaning and connotation, implication and limitation, for a wide range of vocabulary across the many micro-cultures involved. The precursor to developing a survey instrument, or selecting a survey population, or choosing the members of a focus group should involve an analysis of the project itself.

Is the effort worth it? If the evaluation process is a mechanical effort to satisfy some external requirement and will sit dusty and unread in a pile of papers, then the answer is a resounding no. But if the evaluators and developers intend to apply the results to improving the product, then there is no real substitute for taking the time to frame the questions in a way that means something to the people who care.

REFERENCES

Association of Research Libraries. (2000). Association of Research Libraries statistics ranked lists for academic institutions. Retrieved September 27, 2000 from the World Wide Web: http://fisher.lib.Virginia.EDU/newarl/list.html.

Borneman, J. (1992). Belonging in the two Berlins: Kin, state, nation. New York: Cambridge University Press.

Geertz, C. (1988). Works and lives: The anthropologist as author. Stanford, CA: Stanford University Press.

Geertz, C. (1995). After the fact: Two countries, four decades, one anthropologist. Cambridge, MA: Harvard University Press.

Geertz, C. (1956). The development of the Javanese economy: A socio-cultural approach. Cambridge, MA: MIT, Center for International Studies.

Lesk, M. (1997). Practical digital libraries: Books, bytes, and bucks. San Francisco: Morgan Kaufmann.

Mead, M. (1932). Coming of age in Samoa: A psychological study of primitive youth for Western civilization. New York: Morrow.

Nardi, B. A., & O'Day, V. L. (1999). Information ecologies: Using technology with heart. Cambridge, MA: MIT Press.

Nardi, B. A.; Whittaker, S.; & Schwarz, H. (2000). It's not what you know, it's who you know: Work in the information age. First Monday, 5(5). Retrieved September 27, 2000 from the World Wide Web: http://firstmonday.org/issues/issue5_5/nardi/index.html.

National Gallery of the Spoken Word. (1998). Retrieved September 27, 2000 from the World Wide Web: http://www.lib.msu.edu/seadle/Grant_proposals/NGSW.htm.

Schoumantoff, A. (1999). Legends of the American desert: Sojourns in the greater southwest. New York: HarperPerennial.

MICHAEL SEADLE is Digital Services and Copyright Librarian at Michigan State University, and Co-Principal Investigator on the National Gallery of the Spoken Word, a Digital Library Initiative 2 grant. He is also editor of Library Hi Tech. He has written thirty articles, chapters, and books on a wide range of subjects, including German history, computing management, and digitization. Dr. Seadle has a Ph.D. in history from the University of Chicago, a library degree from the University of Michigan, and a decade of experience as a computer professional.

Michael Seade, 100 Library, Michigan State University, East Lansing, MI 48824-1948