Printer Friendly
The Free Library
22,741,889 articles and books

Digital Library Evaluation: Toward an Evolution of Concepts(1).


WHILE THERE WERE MANY EFFORTS IN THE RESEARCH and practices of digital libraries, evaluation was not a conspicuous con·spic·u·ous  
1. Easy to notice; obvious.

2. Attracting attention, as by being unusual or remarkable; noticeable. See Synonyms at noticeable.
 activity. It is well recognized that digital library evaluation is a complex and difficult undertaking. Challenges facing digital library evaluation are enumerated This term is often used in law as equivalent to mentioned specifically, designated, or expressly named or granted; as in speaking of enumerated governmental powers, items of property, or articles in a tariff schedule. . A conceptual framework For the concept in aesthetics and art criticism, see .

A conceptual framework is used in research to outline possible courses of action or to present a preferred approach to a system analysis project.
 for evaluation is suggested. A review of evaluation efforts in research and practice concentrates on derivation derivation, in grammar: see inflection.  of criteria used in evaluation. Essential requirements for evaluation are stated. Discussed are constructs, context, and criteria of digital libraries: What should we evaluate? For what purpose do we evaluate? Who should evaluate? At what level do we evaluate? Upon what criteria do we evaluate? In addition, included are suggestions for adaptation of criteria from related activities. The article is considered as a part of the evolution of concepts for digital library evaluation.


Digital libraries have a short yet turbulent and explosive history. A number of early visionaries Visionaries may refer to:
  • The Visionaries, a hip hop band from Los Angeles
  • , a 13 episode cartoon series, 6 issue comic run and a toyline from the late 1980s
  • Visionaries with Antanas Mockus, a Colombian political party
, such as Licklider (1965), had a notion of libraries in the future being highly innovative and different in structure, processing, and access through heavy applications of technology. But, besides visionary 1. visionary - One who hacks vision, in the sense of an Artificial Intelligence researcher working on the problem of getting computers to "see" things using TV cameras. (There isn't any problem in sending information from a TV camera to a computer.  and futuristic fu·tur·is·tic  
1. Of or relating to the future.

a. Of, characterized by, or expressing a vision of the future: futuristic decor.

 discussions and highly scattered Scattered

Used for listed equity securities. Unconcentrated buy or sell interest.
 research and developmental experimentation, nothing much happened in the next two decades. By the end of the 1980s, digital libraries (under various names) were barely a part of the landscape of librarianship li·brar·i·an  
1. A person who is a specialist in library work.

2. A person who is responsible for a collection of specialized or technical information or materials, such as musical scores or computer documentation.
, information science, or computer science. But just a decade later, by the end of the 1990s, research, practical developments, and general interest in digital libraries exploded ex·plode  
v. ex·plod·ed, ex·plod·ing, ex·plodes

1. To release mechanical, chemical, or nuclear energy by the sudden production of gases in a confined space:
 globally. What a phenomenal decade for work on digital libraries. The accelerated growth of numerous and highly varied efforts related to digital libraries continues unabated un·a·bat·ed  
Sustaining an original intensity or maintaining full force with no decrease: an unabated windstorm; a battle fought with unabated violence.
 in the 2000s.

While the exciting history has yet to be written, Borgman's (1999) discussion of competing visions for digital libraries is a good beginning for understanding the forces and players involved. These competing visions and associated definitions come from several communities that are involved in digital library work. The work of two communities, research and practice, are reviewed below. While they work and proceed mostly independently of each other, they can be considered as two ends of a spectrum, which as yet have not met in the middle. The research community, on one end of the spectrum, asks research questions directed toward future vision or visions of digital libraries, or rather of their various aspects and components, unrestricted by practice. On the other end of the spectrum, the practice community asks developmental and operational questions in real-life real-life  
Actually happening or having happened; not fictional: a documentary with footage of real-life police chases. 
 economic and institutional contexts, restrictions, and possibilities, concentrating on applications on the "market" end of the spectrum.

Large resources and efforts have been expended ex·pend  
tr.v. ex·pend·ed, ex·pend·ing, ex·pends
1. To lay out; spend: expending tax revenues on government operations. See Synonyms at spend.

 on digital library research and practice. There are many efforts, projects, and implementations, not only in the United States United States, officially United States of America, republic (2005 est. pop. 295,734,000), 3,539,227 sq mi (9,166,598 sq km), North America. The United States is the world's third largest country in population and the fourth largest country in area.  but in many other countries and on international levels as well. More are underway. Many exciting things are being done and explored. However, evaluation is more conspicuous by its absence (or just minimal presence) in the vast majority of published work on digital libraries, in either research or practice. So far, evaluation has not kept pace with efforts in digital libraries (or with digital libraries themselves), has not become a part of their integral activity, and has not been even specified as to what it means and how to do it. At this stage of digital library evolution, evaluation in any formal sense (as opposed to anecdotal anecdotal /an·ec·do·tal/ (an?ek-do´t'l) based on case histories rather than on controlled clinical trials.
anecdotal adjective Unsubstantiated; occurring as single or isolated event.
) is being more or less bypassed. True, evaluation has been talked about and implemented in a few instances (as reviewed below), but these are exceptions rather than the rule. Why is that? Some speculations Speculations is an online resource for writers who wish to break into or increase their presence within the science fiction, fantasy, or other speculative fiction subgenres. Speculations has been a Hugo Award nominee seven times. The website is maintained by Kent Brewster.  are:

* Perhaps it is too early in the evolution of digital libraries to attempt evaluation in any formal way. Evaluation at this stage of evolution may be premature and even dangerous because of possible stifling effects.

* At this stage, informal and anecdotal ways of evaluation suffice suf·fice  
v. suf·ficed, suf·fic·ing, suf·fic·es

1. To meet present needs or requirements; be sufficient: These rations will suffice until next week.

* Maybe evaluation is taken to be sufficient on a very basic technical level--the fact that something computes or that an electronic collection is searchable and accessible is sufficient as evaluation in itself.

* From a cynical perspective, we might suggest that the interest in evaluation is suppressed sup·press  
tr.v. sup·pressed, sup·press·ing, sup·press·es
1. To put an end to forcibly; subdue.

2. To curtail or prohibit the activities of.

. Who would want to know about or demonstrate the actual performance?

* On the other hand, perhaps in the pressure of the rapid pace of evolution, the rush to do something and then to rush to something next does not leave time for evaluation.

* And maybe evaluation of digital libraries is so complex that, even when desired, it cannot be accomplished with what we presently know about evaluation. In other words Adv. 1. in other words - otherwise stated; "in other words, we are broke"
put differently
, we might conclude that the conceptual state-of-the-art of digital library evaluation is not sufficiently developed to start with.

While all these speculations may be true to some extent, I believe that the last, the one about the underdeveloped un·der·de·vel·oped
Not adequately or normally developed; immature.
 conceptual nature of evaluation, is actually true. Evaluation of digital libraries is a complex undertaking, and thus it is a conceptual and pragmatic challenge. The main purpose of this discussion is to address various conceptual and theoretical questions about the evaluation of digital libraries and to propose concepts and approaches believed to be appropriate toward their evaluation. The article is considered as a part of the evolution of the concepts for digital library evaluation.


While there are numerous communities interested in digital libraries, the concentration here is, as mentioned, on the research and practice communities as being most closely evaluation bound. Each has a differing interpretation and definition affecting the conceptual nature of evaluation. This translates into specific questions: What is a digital library? What is there to evaluate? What are the criteria? How to apply them in evaluation? Why evaluate digital libraries in the first place?

The distinction (and possible source of tension and lack of communication) between the two communities and approaches has been nicely illustrated by Rusbridge (1998) while contrasting two different approaches to digital libraries--i.e., the U.K. approach in the electronic libraries (eLib) program with the U.S. approach in the Digital Library Initiatives Digital Library Initiative - A project to research digital libraries which aims to provide real collections to real users (high school students, University researchers and students, users in public libraries).  (DLI DLI Department of Land Information (Western Australia)
DLI Donor Lymphocyte Infusion
DLI Defense Language Institute
DLI Durham Light Infantry (Regiment of the British Army) 
   The participants [at digital library conferences in the United States
   reflecting DLI] aimed (properly) to be innovative and free-thinking,
   leaving aside the constraints of existing practice. The results are
   exciting and extraordinarily interesting, but it is very hard to determine
   how many of these ideas might be effectively deployed in real life
   situations. It is notoriously difficult to transfer new technology from
   experiment to practice, but this is clearly harder the more distant the
   experimental context from real life.

      By contrast, the eLib program characterised itself right from the start
   as "development" rather than research.... [The mission of Joint Information
   Systems Committee (JISC) funding the eLib projects] is to stimulate and
   enable the cost effective exploitation of information systems and to
   provide a high quality national network infrastructure for the UK higher
   education and research ... communities; in this context, JISC funds a
   number of development programs aimed at supporting universities by piloting
   the use of appropriate new technologies. Unlike the fundamental research
   characteristics of the NSF and similar agencies, JISC's projects are
   concentrated at the near-market practical application end of the spectrum.
   Both are needed. The eLib work is still research, despite a curious disdain
   for the word in some quarters.


The research community, with most members having a background in computer science, concentrates on developmental research and experimentation in dealing with technology applications in a variety of areas and media, for various communities, and on enabling technologies and networks as an infrastructure for digital libraries. While there is a notion that the research will result in practical applications and in actual digital libraries, the goal is not connected to actual operations but to research. This is an important point to consider because it impinges on evaluation.

In the United States, digital library research is guided, and even defined, through the projects supported by Digital Library Initiatives (DLI). The DLI are funded by a consortium of government agencies under the leadership of the National Science Foundation (NSF NSF - National Science Foundation ). DLI-1 (1994-1998), funded by three agencies, involved six large projects. DLI-2 (1999-2003), funded by eight agencies, involves approximately sixty large and small projects. There are also large research digital library initiatives funded, among others, in the United Kingdom, Germany Germany (jûr`mənē), Ger. Deutschland, officially Federal Republic of Germany, republic (2005 est. pop. 82,431,000), 137,699 sq mi (356,733 sq km). , Japan, Australia, New Zealand New Zealand (zē`lənd), island country (2005 est. pop. 4,035,000), 104,454 sq mi (270,534 sq km), in the S Pacific Ocean, over 1,000 mi (1,600 km) SE of Australia. The capital is Wellington; the largest city and leading port is Auckland. , and regionally by the European Union European Union (EU), name given since the ratification (Nov., 1993) of the Treaty of European Union, or Maastricht Treaty, to the

European Community
. This article concentrates on the efforts in the United States while recognizing the existence of many other efforts in many other countries and regions.

DLIs did not define "digital library." In order to incorporate a wide range of possible approaches and domains, the concept is treated broadly and vaguely. Thus, the projects, particularly in DLI-2, cover a wide range of topics, stretching the possible meaning of "digital library" to, and even beyond the limit of, what can be considered as being "digital" and at the same time recognizable as any kind of a "library" or a part thereof. This is perfectly acceptable for research--frontiers need to be stretched. But, at the same time, it makes evaluation not exactly a possibility to start with. It is not surprising, then, that evaluation is hardly a significant part of DLI efforts.

While formal evaluation was not a big part of DLI-1, three interesting approaches merged. The most notable formal evaluation was done within the Alexandria Digital Library Project (ADL) at the University of California, Santa Barbara History
The predecessor to UCSB, Santa Barbara State College, focused on teacher training, industrial arts, home economics, and foreign languages. Intense lobbying by an interest group in the City of Santa Barbara led by Thomas Storke and Pearl Chase persuaded the State
 (Hill et al., 2000). The approach included a series of user studies involving different user communities and concentrating on different design features as related to their usability How easy something is to use. Both software and Web sites can be tested for usability. Considering how difficult applications are to use and Web sites are to navigate, one would wish that more designers took this seriously. See user interface and usability lab.  and functionality. Some of the results were fed back to improvements of design, "influenc[ing] the Project's implementation goals and priorities." The results served as a base for specifying a "partial list of requirements for new ADL interfaces that came from user evaluation studies." User logs were also studied as a part of the evaluation. The evaluation concentrated on users and their interactions through the interface, with usability and functionality as the main criteria. The usability studies have become one of the more popular ways to approach and implement digital library evaluation (e.g., Buttenfield, 1999). But usability is only one of the possible and needed criteria and approaches.

In the DLI-1 project at the University of California at Berkeley (body, education) University of California at Berkeley - (UCB)

See also Berzerkley, BSD.

Note to British and Commonwealth readers: that's /berk'lee/, not /bark'lee/ as in British Received Pronunciation.
, as part of the evaluation, a series of interviews with intended users were conducted (Schiff, Van House, & Butler, 1997). They focused on situated actions, defined as "[action] performed by specific individuals in specific sociocultural so·ci·o·cul·tur·al  
Of or involving both social and cultural factors.

 context using tools and technologies for a specific purpose." A sociological theory Sociological Theory is a peer-reviewed journal published by Blackwell Publishing for the American Sociological Association. It covers the full range of sociological theory - from ethnomethodology to world systems analysis, from commentaries on the classics to the latest  about the relationship between individual agency and fields of behavioral behavioral

pertaining to behavior.

behavioral disorders
see vice.

behavioral seizure
see psychomotor seizure.
 orientation by Pierre Bourdieu Pierre Bourdieu (August 1, 1930 – January 23, 2002) was an acclaimed French sociologist whose work employed methods drawn from a wide range of disciplines: from philosophy and literary theory to sociology and anthropology.  (1990) was used as a framework. He concluded, "investigating the social setting for which a DL is intended provides us with a rich understanding of the people involved, their relative interest and abilities to act, their opportunities and constraints CONSTRAINTS - A language for solving constraints using value inference.

["CONSTRAINTS: A Language for Expressing Almost-Hierarchical Descriptions", G.J. Sussman et al, Artif Intell 14(1):1-39 (Aug 1980)].
, and their goals." The criteria for the study of users are social environment and user actions. However, it is not clear whether Bourdieu's theory of "habitas" can be immediately applied and used to test digital libraries.

In the DLI-1 project at the University of Illinois University of Illinois may refer to:
  • University of Illinois at Urbana-Champaign (flagship campus)
  • University of Illinois at Chicago
  • University of Illinois at Springfield
  • University of Illinois system
It can also refer to:
, academic researchers studied how readers use scientific journal articles in both print and digital environments--how they "mobilized the work ... as they identify, retrieve, read and use material in articles of interest" (Bishop, 1999, pp. 255-56). The criteria were work and use of retrieved materials by users. In another report by Bishop (1998), the criteria related to access were prominently investigated with results aimed at removing trivial TRIVIAL. Of small importance. It is a rule in equity that a demurrer will lie to a bill on the ground of the triviality of the matter in dispute, as being below the dignity of the court. 4 Bouv. Inst. n. 4237. See Hopk. R. 112; 4 John. Ch. 183; 4 Paige, 364.  and other barriers to access and use.

These three projects, and similar studies of user behavior related to digital libraries or to information in general, provide useful information, as pointed out by Bishop (1999), "[with] implications for user education and digital library system design" (p. 257). But they are really not directly devoted to systematic evaluation. This raises the larger point: User studies, while useful for understanding how people use systems, by themselves are not evaluation even though they may have evaluative implications and they provide important criteria that can be used in evaluation.


The practice community, whose majority resides in operational libraries, concentrates on building operational digital libraries, their maintenance and operations, and providing services to users. The approach is eminently em·i·nent  
1. Towering or standing out above others; prominent: an eminent peak.

2. Of high rank, station, or quality; noteworthy:
 practical, with relatively little research involved. As a result, hundreds, if not thousands, of digital libraries have emerged worldwide, with more becoming operational every day. The efforts are diverse. Many approaches are being used. Numerous types of collections and media are included and processed in many different ways. Several are located in libraries, creating a hybrid library The hybrid library is a term used by librarians to describe libraries containing a mix of traditional print library resources and the growing number of electronic resources.  (combining a traditional and digital library), while others are not bound to libraries at all. The Library of Congress on its Web pages provides an impressive set of links to various digital libraries (starting with and so does the journal D-Lib Magazine D-Lib Magazine is an on-line magazine dedicated to digital library research and development. Content of current and past issues are available free of charge. The publication is financially supported by the Defense Advanced Research Projects Agency (as part of the Digital  ( The American Memory American Memory is an Internet-based archive for public domain image resources, as well as audio, video, and archived Web content. It is published by the Library of Congress. The archive came into existence on October 13, 1994 after $13,000,000 was raised in donations.  Project, pioneered by the Library of Congress, has provided a template (1) A pre-designed document or data file formatted for common purposes such as a fax, invoice or business letter. If the document contains an automated process, such as a word processing macro or spreadsheet formula, then the programming is already written and embedded in the  for many other projects and is also among the earliest such projects to have paid attention to evaluation with criteria of use, usability, and a variety of technical aspects ( and http://memory.

Among the early and longest lasting evaluations of practical digital libraries is the evaluation of the Perseus Project The Perseus Project is a digital library project of Tufts University that assembles digital collections of humanities resources. It is hosted by the Department of Classics. , a corpus of multimedia materials and tools related to the ancient Greek Noun 1. Ancient Greek - the Greek language prior to the Roman Empire
Greek, Hellenic, Hellenic language - the Hellenic branch of the Indo-European family of languages
 world (http:// The mission of Perseus is to provide improved access to primary source materials Noun 1. source materials - publications from which information is obtained
source - a document (or organization) from which information is obtained; "the reporter had two sources for the story"
 related to the needs of students and faculty and to foster greater understanding of culture. The evaluation addressed a set of questions related to learning, teaching, scholarly research in the humanities, and electronic publishing An umbrella term for non-paper publishing, which includes publishing online or on media such as CDs and DVDs.  (Marchionini & Crane, 1994). Four evaluation criteria were identified: (1) learning, (2) teaching, (3) system (performance, interface, electronic publishing), and (4) content (scope, accuracy). The evaluation provided a number of results that were summarized in four categories: amplification amplification /am·pli·fi·ca·tion/ (33000) (am?pli-fi-ka´shun) the process of making larger, such as the increase of an auditory stimulus, as a means of improving its perception.  and augmentation AUGMENTATION, old English law. The name of a court erected by Henry VIII., which was invested with the power of determining suits and controversies relating to monasteries and abbey lands.  of learning, physical infrastructure, conceptual infrastructure, and systemic systemic /sys·tem·ic/ (sis-tem´ik) pertaining to or affecting the body as a whole.

1. Of or relating to a system.

 change to the field. This is still a model evaluation project for digital libraries.

PEAK (Pricing Electronic Access to Knowledge) is one of the more interesting projects that involve,; both observation of use and evaluation of a variety of aspects, particularly including economic factors (Bonn, Lougee, Mackie-Mason, & Riveros, 1999; Mackie-Mason, Riveros, Bonn, & Lougee, 1999). It is unique in that it involves a publisher of electronic journals, Elsevier Science, and about a dozen libraries. (A project, TULIP, also done by Elsevier and a number of universities, preceded PEAK.) Criteria for evaluation included access (different types of access to journals were offered to different groups), pricing (different models), and revenues and costs. This project extended evaluation criteria and measures to economic factors or efficiency evaluation.

The Museum Educational Site Licensing (MESL MESL Museum Educational Site Licensing
MESL Museum of Eastern Shore Life (Centreville, MD)
MESL Mission Essential Subsystem List
MESL Mission Event Synchronization List
MESL Men's Evangelical Softball League
) Project was a collaboration Working together on a project. See collaborative software.  of seven collecting institutions and seven universities, defining the terms and conditions for the educational use of digitized museum images and related information. The MESL implementation at Cornell Cornell

named after New York State Veterinary College at Cornell University, NY, USA.

Cornell alternative-month accelerated lambing system
enables each ewe to lamb three times in every 2 years.
, as a separate digital library, has been conducted under the auspices aus·pi·ces 1  
Plural of auspex.

Noun, pl

under the auspices of with the support and approval of [Latin auspicium augury from birds]

 of the university's Digital Access Coalition. A report describes the implementations and some evaluation (Cornell, 1999). The approach is impressionistic--many questions have been asked of users, designers, developers, and operators to obtain evaluative impressions and assessments. Criteria in questions include: functionality--i.e., browsing See browse. , searching, difficulty, usage, experiences, training needs, integration into other campus services, preparation of source materials for inclusion, fields indexed, server performance, system security and authentication (1) Verifying the integrity of a transmitted message. See message integrity, e-mail authentication and MAC.

(2) Verifying the identity of a user logging into a network.
, ongoing support needed or desired, technical development, physical infrastructure, costs, time, and skills. The evaluation was not formal, but it is interesting if for nothing else than for the breadth of criteria included.

Since 1995, the Human-Computer Interaction Human-computer interaction

An interdisciplinary field focused on the interactions between human users and computer systems, including the user interface and the underlying processes which produce the interactions.
 Group at Cornell University Cornell University, mainly at Ithaca, N.Y.; with land-grant, state, and private support; coeducational; chartered 1865, opened 1868. It was named for Ezra Cornell, who donated $500,000 and a tract of land. With the help of state senator Andrew D.  has conducted research or evaluation studies of a number of prototype efforts to build digital collections in museums and libraries (Jones, Gay, & Rieger, 1999). In that paper, they summarized five studies. The criteria used revolve around Verb 1. revolve around - center upon; "Her entire attention centered on her children"; "Our day revolved around our work"
center, center on, concentrate on, focus on, revolve about
 "backstage" concerns or representation; legal issues ("e.g., metadata (1) (meta-data) Data that describes other data. The term may refer to detailed compilations such as data dictionaries and repositories that provide a substantial amount of information about each data element. , copyright and intellectual property issues"); collection maintenance and access ("e.g., decisions regarding collection scope and the maintenance of a consistent quality and fidelity of digital records"); and usability ("e.g., user skill levels and expectations, and the use of collections in formal and informal educational settings"). The methods used in these evaluations are not clear--i.e., to what degree were they formal or informal? But a number of conclusions were drawn. Among them: "Effective digital collections are complex sociotechnical systems: An effective collection requires consistent and simultaneous attention to a variety of social, organizational, administrative, and technical concerns" (Jones, Gay, & Reiger, 1999). A number of other authors came to the same conclusion, illustrating a model of digital libraries that involves a wide range of levels, as suggested later in this discussion.

Kilker and Gay (1998), in providing a framework for evaluation and applying that framework to a case study, expressed ideas that were similar to those of other studies at Cornell. The framework was the Social Construction of Technology (SCOT) theory, where the concentration is on examining varied conceptions held by "relevant social groups" involved in technology development and use. The approach is presented as an alternative to system and user-centered frameworks for study and evaluation. It recognizes that different audiences associated with a digital library (from designers to different groups of users) have different interpretations; they evaluate a digital library differently and use a different terminology. The criteria are: relevant social groups, interpretive in·ter·pre·tive   also in·ter·pre·ta·tive
Relating to or marked by interpretation; explanatory.

in·terpre·tive·ly adv.
 flexibility (capabilities, responses), and mediation mediation, in law, type of intervention in which the disputing parties accept the offer of a third party to recommend a solution for their controversy. Mediation has long been a part of international law, frequently involving the use of an international commission, .

In a joint international undertaking, the National Science Foundation (primary sponsor of DLI research in the United States) and Joint Information Systems Committee (JISC JISC Joint Information Systems Committee (UK)
JISC Japan Industrial Standards Committee
JISC Joint Industry Safety Committee
) (primary sponsor of the eLib program in the United Kingdom) developed in 1999 a joint initiative or, as they called it, "a hybrid process ... in order to bring together the best elements of the styles of the two funding bodies A funding body is an organisation that provides funds in the form of research grants or scholarships. Research Councils
Research Councils are funding bodies that are government-funded agencies engaged in the support of research in different disciplines and
." The idea is to fuse the two approaches, where the objective of JIST JIST Juneau Information Service Technology
JIST Joint Imagery Soft-copy Trainer
JIST JSF Integrated Subsystems Technology
JIST Joint Interagency Support Team
JIST Maybe you're looking for the word 'Gist' meaning the central idea?
 is "development of content or new technologies that would be widely applicable and not just of benefit to the participating institutions," while NSF's objective is "new research in the area of digital libraries, and the presence of new scientific ideas and methods." The efforts of funded projects are geared toward criteria showing "ability of the international partners to work together," and to combine research with practical development (Wiseman, Rusbridge, & Griffin, 1999). These criteria differ from others applied in either research or practice. However, from examination of abbreviated proposals, the funded projects under this international initiative have little in the way of evaluation built in.

There is still another practical concern that closely relates to evaluation. For over 100 years, ever since Melvil Dewey, library collections have been built and managed in relation to some established standards and policies. These provided criteria for traditional evaluation of collections. Not surprisingly, a number of libraries and library-related consortia that are in the process of developing or acquiring digital collections have also undertaken establishment of standards and policies for such collections. Okerson (1999) provides links to more than thirty library sites announcing their standards or policies for digital or electronic collections. In turn, these are incorporating, directly or indirectly, criteria for evaluation of digital collections and raising significant issues about the standards themselves and their use in evaluation. Most of the criteria incorporated are derived from traditional library collection criteria, as enumerated below--and they fit well. But, slowly, some additional criteria are emerging. Among them are strategic significance and availability of other distributed sources, such as found on the Internet Internet

Publicly accessible computer network connecting many smaller networks from around the world. It grew out of a U.S. Defense Department program called ARPANET (Advanced Research Projects Agency Network), established in 1969 with connections between computers at the
 or databases in the organization (e.g., on campus). Organizations not directly connected with libraries are also concerned with policies for digital collections and databases; good examples are the elaborate policies and criteria established by the Arts and Humanities Data Service The Arts and Humanities Data Service (AHDS) is a United Kingdom national service aiding the discovery, creation and preservation of digital resources in and for research, teaching and learning in the arts and humanities. It was established in 1996.  (United Kingdom) (Beagrie & Greenstein, 1998). An important issue there is validation See validate.

validation - The stage in the software life-cycle at the end of the development process where software is evaluated to ensure that it complies with the requirements.
 of sources included. It turns out that validation is a key problem in the use of Internet resources in general. The efforts of libraries to provide standards and criteria for their digital holdings that are then available (generally or to restricted audiences) over the Internet are establishing trust, validity, and authority for their own resources on the Internet, thus promoting access with user confidence, a highly important thing on the otherwise value-neutral Internet.

To summarize sum·ma·rize  
intr. & tr.v. sum·ma·rized, sum·ma·riz·ing, sum·ma·riz·es
To make a summary or make a summary of.

: this review includes representative efforts, primarily to illustrate criteria used. It does not claim to cover the entire subject. But evaluation coverage generally is not large; few other evaluations were found in either research or practice. This illustrates the point that there is a dearth of evaluation efforts in comparison to all efforts related to digital libraries.


The general questions in any and all evaluations are: Why evaluate? What to evaluate? How to evaluate? For whom to evaluate? There are many approaches to evaluation and to answering these questions. We must fully recognize the appropriateness of different approaches for different evaluation goals and audiences. For instance, the ethnographic eth·nog·ra·phy  
The branch of anthropology that deals with the scientific description of specific human cultures.

 approach is highly appropriate for gaining a broad understanding of the role and effects of a practice or a construct in a wider social or organizational framework. The sociological approach is appropriate in illuminating il·lu·mi·nate  
v. il·lu·mi·nat·ed, il·lu·mi·nat·ing, il·lu·mi·nates
1. To provide or brighten with light.

2. To decorate or hang with lights.

 the social forces and effects. The economic approach is appropriate in accounting for economic factors, the political science approach for policy and political factors, and so on. Clearly, every approach has strengths and weaknesses; there is no one "best" approach. It is naive naive - Untutored in the perversities of some particular program or system; one who still tries to do things in an intuitive way, rather than the right way (in really good designs these coincide, but most designs aren't "really good" in the appropriate sense).  to argue for a predominance pre·dom·i·nance   also pre·dom·i·nan·cy
The state or quality of being predominant; preponderance.

Noun 1. predominance - the state of being predominant over others
predomination, prepotency
 of any given approach. The answer to the first question as to why to evaluate should serve as a base for selection of an appropriate approach or approaches.

However, here the concentration is on the systems approach only as the most widely practiced or suggested approach for evaluation of all kinds of information systems, including digital libraries, fully recognizing both its strengths and limitations. At the outset, the basic assumption of all systems approaches is that evaluation deals with some aspect of performance. Thus, the general why of evaluation deals with performance to start with and goes on from there to define more specific goals and choices as discussed under the context of evaluation below.

To establish a common vocabulary and concepts, a few standard definitions follow. A system can be considered as a set of elements in interaction. A human-made system, such as a digital library, has an added aspect: it has certain objective (s). The elements, or components, interact to perform certain functions or processes to achieve given objectives. Furthermore, any system (digital libraries included) exists in an environment, or more precisely in a set of environments (which can also be thought of as systems, and some may think of this as contexts), and interacts with its environments. It is difficult, and even arbitrary, to set the boundaries of a system. In the evaluation of digital libraries, as in the evaluation of any system or process, these difficult questions arise that clearly affect the results: Where does a digital library under evaluation begin to be evaluated? Where does it end? What are the boundaries? What to include? What to exclude? On what environment or context to concentrate? This provides the questions for determining the construct of digital libraries, as discussed below.

In this context, evaluation means an appraisal of the performance or functioning of a system, or part thereof, in relation to some objective(s). The performance can be evaluated as to:

* effectiveness: How well does a system (or any of its parts) perform that for which it was designed?

* efficiency: At what cost (costs could be financial or involve time or effort)?

* a combination of these two (i.e., cost-effectiveness cost-effectiveness

pertaining to cost-effective.

cost-effectiveness analysis
a comparison of the relative cost-efficiencies of two or more ways of performing a task or achieving an objective.

An evaluation has to specify which of these will be evaluated. This discussion will primarily involve the evaluation of effectiveness with a realization that, during any evaluation of efficiency, cost-effectiveness can be involved as well. This sets the questions of the criteria of evaluation for digital libraries as discussed below.

As in all systems, objectives occur in hierarchies, and there may be several hierarchies representing different levels--sometimes even in conflict. While the objectives may be explicitly stated or implicitly derived or assumed, they have to be reflected in an evaluation. Evaluation is not one fixed thing. For the same system, evaluation can be done on different levels, in relation to different choices of objectives, using a variety of methods, and it can be oriented o·ri·ent  
1. Orient The countries of Asia, especially of eastern Asia.

a. The luster characteristic of a pearl of high quality.

b. A pearl having exceptional luster.

 toward different goals and audiences.

To be considered an evaluation, it has to meet certain requirements. It must involve selections and decisions related to the:

1. Construct for evaluation. What to evaluate? What is actually meant by a digital library? What is encompassed? What elements (components, parts, processes) to involve in evaluation?

2. Context of evaluation. Selection of a goal, framework, viewpoint, or level(s) of evaluation. What is the level of evaluation? What is critical for a selected level? Ultimately, what objective(s) to select for that level?

3. Criteria reflecting performance as related to selected objectives. What parameters of performance to concentrate on? What dimension or characteristic to evaluate?

4. Measures reflecting selected criteria to record the performance. What specific measure(s) to use for a given criterion?

5. Methodology for doing evaluation. What measuring instruments to use? What samples? What procedures to use for data collection? For data analysis?

A clear specification on each of these is a requirement for any evaluation of digital libraries. Unfortunately, it is not as yet entirely clear what is to be specified in each of these five elements five elements, fire, water, earth, wood, and metal; in Chinese medicine, each of these five components is used to organize phenomena for use in clinical applications. Each of the elements corresponds to a specific function (i.e.
. No agreement exists on criteria, measures, and methodologies for digital library evaluation, or even on the "big picture," the construct and context of evaluation. The evaluation of digital libraries is still in a formative formative /for·ma·tive/ (for´mah-tiv) concerned in the origination and development of an organism, part, or tissue.  stage. Concepts have to be clarified first. This is the fundamental challenge for digital library evaluation.

A clarification is needed as to what does not fall in the realm of evaluation, even though it could be related to evaluation. By themselves, measurement, collection of statistics, or specification of metrics metrics Managed care A popular term for standards by which the quality of a product, service, or outcome of a particular form of Pt management is evaluated. See TQM.  for digital libraries are not evaluation--they are quantitative or qualitative characterizations. Observation by itself, such as observing user behavior in the use of a digital library, is not evaluation. Assessing user needs by itself is not evaluation, and neither is relating those needs to design. However, these can be linked to evaluation if, and only if, they are connected to some specified performance which includes all five requirements enumerated earlier.

A related view of evaluation is expressed by Marchionini, Plaisant, and Komlodi (in press):
   Evaluation of a digital library may serve many purposes ranging from
   understanding basic phenomena (e.g., human information-seeking behavior) to
   assessing the effectiveness of a specific design to insuring sufficient
   return on investment. Human-centered evaluation serves many stakeholders
   ranging from specific users and librarians to various groups to society in
   general. Additionally, evaluation may target different goals ranging from
   increased learning and improved research to improved dissemination to
   bottom line profits. Each of the evaluation goals may also have a set of
   measures and data collection methods. Finally, the evaluation must have a
   temporal component that can range from very short terms to generations. (p.


What is there to evaluate? A simplistic sim·plism  
The tendency to oversimplify an issue or a problem by ignoring complexities or complications.

[French simplisme, from simple, simple, from Old French; see simple
 answer is that whatever is called a "digital library" project is therefore considered a digital library, thus a construct candidate for evaluation. (This is derived from a certain philosophical stance whose metaphor is "Physics [or whatever field] is what a physicist does".) This is a pragmatic approach that has been at times applied to modeling a construct of a digital library; to some extent it even works. But a more formal approach to defining or modeling the construct is needed in order to develop generalizations to and from evaluations.

Because digital libraries are related to physical libraries and may perform a number of similar functions, but in relation to a digital and distributed collection, the modeling and evaluation of digital libraries may, to some extent, parallel those related to physical libraries--at least initially. But (and this is a very important "but") digital libraries are also quite different and, in some functions, as for example in distribution and access, completely different from physical libraries. Thus, digital libraries also require additional and new approaches to modeling of their constructs and thus to evaluation as well. Also, a digital library is much more than a collection of digitized texts and other objects. The challenge at the beginning of digital library evaluation is developing and applying these new modeling concepts to the specifics of what is meant by, and incorporated in, a "digital library."

As mentioned, in the research community, "digital library" has not been defined. The closest to the definition applicable to the approaches taken by the research community is the one given by Lesk (1997) in the first textbook textbook Informatics A treatise on a particular subject. See Bible.  on the topic:
   digital libraries are organized collections of digital information. They
   combine the structure and gathering of information, which libraries and
   archives have always done, with the digital representation that computers
   have made possible. (emphasis added)

The emphasized elements in the definition represent constructs that could and should enter into evaluation, answering the question at the start of this section. The question should be raised: Is this enough? I do not think so.

Borgman (1999) provides a more complex definition (including an extensive discussion) of digital libraries, a definition that may be considered as a bridge between the research community definition above and practical community definition below:

1. Digital libraries are a set of electronic resources and associated technical capabilities for creating searching, and using information ... they are an extension and enhancement of information storage and retrieval information storage and retrieval, the systematic process of collecting and cataloging data so that they can be located and displayed on request. Computers and data processing techniques have made possible the high-speed, selective retrieval of large amounts of  systems that manipulate manipulate

To cause a security to sell at an artificial price. Although investment bankers are permitted to manipulate temporarily the stock they underwrite, most other forms of manipulation are illegal.
 digital data in any medium.... The content of digital libraries includes data, [and] metadata....

2. Digital libraries are constructed, collected, and organized, by (and for) a community of users, and their functional capabilities support the information needs and uses of that community. (p. 230)

In this definition, the elements in the construct subject or candidates for evaluation are:

* electronic resources--digital data in any medium;

* technical capabilities for creating, searching, and using information;

* information retrieval information retrieval

Recovery of information, especially in a database stored in a computer. Two main approaches are matching words in the query against the database index (keyword searching) and traversing the database using hypertext or hypermedia links.

* metadata; and

* community of users--their information needs and uses.

In a newer text, Arms (2000) provides what he calls an "informal definition": "a digital library is a managed collection of information, with associated services, where the information is stored in digital formats and accessible over a network. The crucial part of this definition is that the information is managed" (p. 2, emphasis added). In this construct, the subjects for evaluation are italicized. The critical element added here is the aspect of management of the collection and information.

In the United States, the Digital Library Federation (DLF DLF Digital Library Federation
DLF Digital Library Federation (Washington, DC)
DLF Development Loan Fund
DLF Distribution Loss Factor
DLF Det Liberale Folkeparti (Norwegian political party) 
) is an organization of research libraries and various national institutions formed in 1995. The stated goal of DLF is "to establish the conditions necessary for the creation, maintenance, expansion, and preservation of a distributed collection of digital materials accessible to scholars and the wider public" (DLF, September 17, 1999). The organization represents the practical community. After considerable work, DLF agreed on a "working definition of a digital library" representing a definition of the practice community:
   Digital libraries are organizations that provide the resources, including
   the specialized staff, to select, structure, offer intellectual access to,
   interpret, distribute, preserve the integrity of, and ensure the
   persistence over time of collections of digital works so that they are
   readily and economically available for use by a defined community or set of
   communities. (DLF, April 21, 1999)

This definition and conception is quite different from the one provided by Lesk (1997), Arms (2000), and even by Borgman (1999). Here the emphasis is on an organizational or institutional setting for the collection of digital works and aspects related to its functioning in the larger context of service, which specifically involves these elements in the construct subject or candidates for evaluation:

* professional staff;

* collection of digital works;

* selection, structure, and access;

* interpretation and distribution;

* preservation; and

* use and economic availability for a defined community.

Let us attempt an integration. In a general way, the constructs or elements for evaluation of digital libraries are:

* digital collections, resources;

* selection, gathering, holdings, media;

* distribution, connections, links;

* organization, structure, storage;

* interpretation, representation, metadata;

* management;

* preservation, persistence (1) In a CRT, the time a phosphor dot remains illuminated after being energized. Long-persistence phosphors reduce flicker, but generate ghost-like images that linger on screen for a fraction of a second. ;

* access;

* physical networks;

* distribution;

* interfaces, interaction;

* search, retrieval;

* services;

* availability;

* range of available services--e.g., dissemination dissemination Medtalk The spread of a pernicious process–eg, CA, acute infection Oncology Metastasis, see there , delivery;

* assistance, referral;

* use, users, communities;

* security, privacy, policies, legal aspects, licenses;

* management, operations, staff;

* costs, economics; and

* integration, cooperation with other resources, libraries, or services.

An evaluation of a digital library, either in research or practice, could select what to evaluate from these elements. In other words, an evaluation must specify clearly what elements are evaluated with full recognition of the emphasis on what is included and what is excluded. Every evaluation leaves something out. With the present state of knowledge, no evaluation can cover even the majority of elements involved in a digital library, nor can it pretend to do so. Thus, there is no "evaluation of digital libraries." Possibly, there is only an evaluation of some of the elements in their construct.


Any evaluation is a tuplet between a selected element to be evaluated and a selected element of its performance. This leads to selection of a level of evaluation: What to concentrate on? Digital libraries, like other systems, can be viewed, and thus evaluated, from a number of standpoints or levels. Each of these levels can be translated into a goal for evaluation.

A big dilemma and difficulty in evaluation is the selection of the level of objectives to address. Let us divide objectives, and thus evaluations, of a technical computer-based system, such as a digital library, into seven general classes or levels (of course, they are not mutually exclusive Adj. 1. mutually exclusive - unable to be both true at the same time

incompatible - not compatible; "incompatible personalities"; "incompatible colors"
). The first three are more user-centered and the last three more system-centered with an interface in between. The performance questions for each level are indicated.


Social level. How well does a digital library support the needs and demands, roles, and practices of a society or community? This can be very hard to evaluate due to the diverse objectives of the society or community. Many complex variables are involved.

Institutional. How well does a digital library support the institutional or organizational mission and objectives? How well does it integrate with other institutional resources? This is tied to institutional organizational objectives--also hard to evaluate for similar reasons.

Individual. How well does a digital library (or given services) support information needs, tasks, activities of people as individual users or groups of users with some strong commonalties? It turns out that most evaluations tend to be on that level, probably because it is most direct and easiest to evaluate, though differences in perceptions can prove troublesome, and it is not always easy to generalize generalize /gen·er·al·ize/ (-iz)
1. to spread throughout the body, as when local disease becomes systemic.

2. to form a general principle; to reason inductively.
 to a larger population.

Interface. How well does a given interface provide and support access, searching, navigation, browsing, and interaction with a digital library? Questions can be asked in either the user or system direction or in both directions.

System Centered

Engineering. How well do hardware, networks, and related configurations perform? These questions yield more replicable measures and are more easily generalizable gen·er·al·ize  
v. gen·er·al·ized, gen·er·al·iz·ing, gen·er·al·iz·es
a. To reduce to a general form, class, or law.

b. To render indefinite or unspecific.

 than many user-centered approaches.

Processing. How well do procedures, techniques, algorithms The following is a list of the algorithms described in Wikipedia. See also the list of data structures, list of algorithm general topics and list of terms relating to algorithms and data structures. , operations, and so on perform? These are also very systematic, though there may be variation due to differences in configuration, capacity, and other system variables.

Content. How well is the collection or information resources (1) The data and information assets of an organization, department or unit. See data administration.

(2) Another name for the Information Systems (IS) or Information Technology (IT) department. See IT.
 selected, represented, organized, structured, and managed? Although this is also fairly systematic, the related questions are how well, for whom, and for what purpose?

Moreover, as mentioned, not only effectiveness but also efficiency or cost-effectiveness questions can be asked and contrasted at each level. Evaluation on one level rarely, if ever, answers questions from another. For instance, evaluations of engineering or processing aspects of digital libraries say little about questions arising in the evaluation of use. In real-life operations and applications of digital libraries, a number of levels are closely connected, but evaluations of digital libraries are not. As yet, digital libraries are not evaluated on more than one level. This isolation of levels of evaluation could be considered a further and greater challenge for all digital library evaluations. In addition, as a rule, many systems are used in ways that their designers never intended.


Criteria for each level have to be determined. So far there is little agreement as to what these criteria should be. In the evaluations reviewed above, a level was explicitly or implicitly chosen, and with it a set of criteria was used as enumerated. The level chosen for evaluation most often was the individual level, as defined and, among the criteria, the most prominent was usability.

Marchionini, Plaisant, and Komlodi (in press), at the outset of a chapter that, among other things, addresses design and evaluation of digital libraries, state:
   Digital libraries (DL) serve communities of people and are created and
   maintained by and for people. People and their information needs are
   central to all libraries, digital or otherwise. All efforts to design,
   implement, and evaluate digital libraries must be rooted in the information
   needs, characteristics, and contexts of the people who will or may use
   those libraries. (p. 1, emphasis in the original)

In this concept, evaluation is squarely square·ly  
1. Mathematics At right angles: sawed the beam squarely.

2. In a square shape.

 placed in the realm of user-centered levels, with an implicit, if not explicit, absence of system-centered levels. I disagree with Verb 1. disagree with - not be very easily digestible; "Spicy food disagrees with some people"
hurt - give trouble or pain to; "This exercise will hurt your back"
 the concept that evaluation must or should "a priori a priori

In epistemology, knowledge that is independent of all particular experiences, as opposed to a posteriori (or empirical) knowledge, which derives from experience.
" be based on any one or a set of given levels, be they user- or system-centered. Evaluation can and should be performed at different levels, involving different objectives and related criteria. This issue has been visited, and even vehemently argued, a number of times in the debates about information retrieval (IR) design and evaluation. The conclusion about approaches to IR design and evaluation is valid for digital libraries as well:
   But the issue is not whether we should have systems--OR human-centered
   approaches. The issue is even less of human--VERSUS systems-centered. The
   issue is how to make human--AND systems-centered approaches work together.
   (Saracevic, 1999, p. 1058, emphasis in the original)

For each of the levels, criteria have to be developed and applied. For instance, there is nothing wrong in developing criteria for evaluation of the content level in relation to the collection and asking questions such as: How well does a given collection represent that which exists in a given domain or medium? How timely is it? How well is it represented according to according to
1. As stated or indicated by; on the authority of: according to historians.

2. In keeping with: according to instructions.

 some standard? The last question relates a digital library collection to some standards. These and similar evaluative questions involve just that level, and they are important for assessing a given collection by itself. Thus, not everything has to or should be centered in any one level or a given set of levels.


A number of criteria used in the evaluation of digital libraries were enumerated. Next, suggestions are made about criteria that have been used in practice in related enterprises and that can be considered for adapting into criteria for digital library evaluation.

Libraries, information retrieval systems, and human-computer interfaces (software, hardware) Human-Computer Interface - (HCI) Any software or hardware that allows a user to interact with a computer. Examples are WIMP, command-line interpreter, or virtual reality.

See also Human-Computer Interaction.
 have been evaluated for a long time using numerous criteria. A good number of evaluation criteria for libraries were summarized by Lancaster (1993), for library and information services See Information Systems.  by Saracevic and Kantor (1997), for IR systems by Su (1992), and for interfaces by Shneiderman (1998). Buttenfield (1999) provides a framework for usability evaluation and criteria. From these and other sources, here is a short list of criteria that could, and even must, be adapted for digital libraries:

Traditional library criteria

* collection: purpose, subject, scope, authority, coverage, currency, audience, cost, format, treatment, preservation, persistence;

* information: accuracy, appropriateness, links, representation, uniqueness, comparability, presentation, timeliness, ownership;

* use: accessibility, availability, searchability, usability; and

* standards for a number of elements and processes.

Traditional IR criteria

* relevance (leads to measures of precision and recall);

* satisfaction, success; and

* index, search, output features.

Traditional human-computer interaction/interfaces criteria

* usability, functionality, effort;

* task appropriateness, failures;

* connectivity, reliability;

* design features;

* navigation, browsing; and

* services, help.


Digital libraries have exploded onto the scene. Numerous research and practical efforts and large resources are expended on digital library research and practice. Evaluation is not, by and large, a part of these efforts. With few notable exceptions in either research or practice, digital library evaluation is not conspicuous. Despite these exceptions, digital library evaluation has yet to penetrate research, practice, or even debate. But it must be recognized that digital library evaluation is a complex and difficult undertaking. This article discusses the challenges facing digital library evaluation and suggests a conceptual framework for evaluation derived from the systems approach. Much more has to be specified and agreed upon Adj. 1. agreed upon - constituted or contracted by stipulation or agreement; "stipulatory obligations"

noncontroversial, uncontroversial - not likely to arouse controversy
 before digital library evaluation can be carried out in a consistent manner, a manner that would allow even for comparisons.

A significant point has been made in the opening statement on the Web page of the Digital Library Federation (1999):
   One of the great accomplishments of traditional libraries is that they are
   organized along similar lines. The individual who knows how to use one
   library in this country is likely to be able to use any other. Users have
   come to take this uniformity for granted in the print environment, but it
   is far from the norm in the digital environment.

      Digital resources now available through global networks are anything but
   organized. If digital collections created or stored at one library are to
   be available to others, there must be general agreement about the
   requirements for systems architecture, metadata, indexing, and retrieval.
   The development and adoption of common standards will require significant
   additional effort and exploration.

The evaluation of digital libraries should also be looking at, and contributing to, the gaining of uniformity for access and use across the landscape of digital libraries, which involves evaluation across a number of digital libraries and not only single efforts. While it is way too early to set formal standards for digital libraries and thereby freeze innovation, it is not too early to think about evaluation of factors and features contributing to uniformity as an additional criterion. A further and critical issue for evaluation is persistence. An important feature of many traditional libraries is that their collections are preserved over time--they persist. An important feature of digital collections is a potential lack of persistence. Libraries have no control whatsoever over persistence of digital journals, indexes, and the like for which they have licensed access for a time. Publishers may go out of business, as many do. or they may change direction into some other line, as also many do, and thus the sources under their control will vanish. Digital journals, such as D-Lib Magazine copiously co·pi·ous  
1. Yielding or containing plenty; affording ample supply: a copious harvest. See Synonyms at plentiful.

 cited here, may vanish after their funding runs out. Persistence may become one of the most important criteria for digital libraries.

Even if there is no visible movement in the evolution of digital library evaluation on a formal level, an informal evaluation of digital library efforts will proceed by funders, users, the public, peers, technologists, experts, lay people, and anybody that is involved with the results of digital library research or practice in any way. Such informal evaluations can be valid and reliable but can also stray Stray

(1) Not a member of the participating party in the trade at hand; (2) not a meaningful indication of a customer's desire to take a sizable position or be involved in a stock.
 in significant ways and create erroneous erroneous adj. 1) in error, wrong. 2) not according to established law, particularly in a legal decision or court ruling.  perceptions and expectations of digital libraries. Thus, it is imperative that efforts in formal evaluation of digital libraries be enlarged and become an integral part of all research and practice no matter what the challenge.

After all this is said of evaluation, a larger set of questions loom loom, frame or machine used for weaving; there is evidence that the loom has been in use since 4400 B.C.

Modern looms are of two types, those with a shuttle (the part that carries the weft through the shed) and those without; the latter draw the weft from a
, questions to which I alluded to in the introductory comments: "At this early stage of digital library evolution is it too early to concentrate on evaluation? Could early evaluation stifle innovation? Could it lead into different directions, such as concentrating on minutia mi·nu·ti·a  
n. pl. mi·nu·ti·ae
A small or trivial detail: "the minutiae of experimental and mathematical procedure" Frederick Turner.
 of that which can be measured over the bigger picture? Could premature evaluation turn contra-productive?"

If evaluation is taken rigidly, the answer to all of these questions is "Yes." But if taken in the spirit of evolution of digital libraries, then their evaluation should also be taken as an evolutionary enterprise. Evolution of evaluation should be treated as a necessary part of the larger evolution of digital libraries and, as that larger evolution, it will have a part that ends in blind alleys blind alley
1. An alley or passage that is closed at one end.

2. A mistaken, unproductive undertaking.

blind alley

1. an alley open at one end only

 and it is hoped a much larger part that leads to successes. But, it is never too early to start thinking about it and to go on clarifying evaluation concepts and doing evaluation experiments. This article has been written in that spirit.

The ultimate evaluation of digital libraries will be in relation to the transformation of their context, the same as of evaluation of libraries throughout history. Digital libraries provide for an interaction among people, human knowledge, organizations, and technology. The ultimate question for evaluation is: How are digital libraries transforming research, education, learning, and living? At this stage, we don't have the answers, but we have indications that significant transformations are indeed taking place.


(1) This paper is substantially based on Saracevic, T., & Covi, L. (2000). Challenges to digital library evaluation. Proceedings of the American Society for Information Science, 37.


Arms, W. Y. (2000). Digital libraries. Cambridge, MA: MIT MIT - Massachusetts Institute of Technology  Press.

Beagrie, N., & Greenstein, D. (1998). Managing digital collections: Arts and Humanities Data Service (AHDS AHDS Arts and Humanities Data Service (UK)
AHDS Allan-Herndon-Dudley Syndrome
) policies, standards and practices: Evaluation criteria for evaluation. Retrieved September 29, 2000 from the World Wide Web:

Bishop, A. P. (1998). Measuring access, use, and success in digital libraries. Journal of Electronic Publishing, 4(2). Retrieved September 29, 2000 from the World Wide Web:

Bishop, A. P. (1999). Document structure and digital libraries: How researchers mobilize mo·bi·lize
1. To make mobile or capable of movement.

2. To restore the power of motion to a joint.

3. To release into the body, as glycogen from the liver.
 information in journal articles. Information Processing information processing: see data processing.
information processing

Acquisition, recording, organization, retrieval, display, and dissemination of information. Today the term usually refers to computer-based operations.
 & Management, 35(3), 255-279.

Bonn, M. S.; Lougee, W. P.; Mackie-Mason, J. K; &. Riveros, J. F. (1999). A report on the PEAK experiment: Context and design. D-Lib Magazine, 5(6). Retrieved September 29, 2000 from the World Wide Web:

Bourdieu, P. (1990). The logic of practice. Stanford, CA: Stanford University Stanford University, at Stanford, Calif.; coeducational; chartered 1885, opened 1891 as Leland Stanford Junior Univ. (still the legal name). The original campus was designed by Frederick Law Olmsted. David Starr Jordan was its first president.  Press.

Borgman, C. L. (1999). What are digital libraries? Competing visions. Information Processing & Management, 35(3), 227-243.

Buttenfield, B. (1999). Usability evaluation of digital libraries. Science & Technology Libraries, 17(3/4), 39-59.

Cornell University. (1999). MESL technical report. Retrieved September 29, 2000 from the World Wide Web:

Digital Library Federation. (1999, April 21). A working definition of digital library. Retrieved September 29, 2000 from the World Wide Web:

Digital Library Federation (1999, September 17). DLFHome. Retrieved September 29, 2000 from the World Wide Web:

Hill, L. L.; Carver carver /car·ver/ (kahr´ver) a tool for producing anatomic form in artificial teeth and dental restorations.
carver (carving instrument),
, L.; Larsgaard, M.; Dolin, R.; Smith, T. R.; Frew, J.; & Rae, M.A. (2000). Alexandria Digital Library: User evaluation studies and system design. Journal of the American Society for Information Science, 51(3), 246-259.

Jones, M. L. W.; Gay, G. K.; & Rieger R. H. (1999). Project Soup: Comparing evaluations of digital collection efforts. D-Lib Magazine, 5(11). Retrieved September 29, 2000 from the World Wide Web:

Kilker, J., & Gay. G. (1998). The social construction of a digital library: A case study examining implications for evaluation. Information Technology and Libraries, 17(2), 60-70.

Lancaster, F. W. (1993). If you want to evaluate your library ... (2d ed.). Urbana-Champaign: University of Illinois, Graduate School of Library and Information Science A School of Library and Information Science (SLIS) is a university-based institution that provides a Master's degree or other advanced degrees associated with Library science, Information Science, or a combination of the two. .

Lesk, M. E. (1997). Practical digital libraries: Books, bytes, and bucks. San Francisco San Francisco (săn frănsĭs`kō), city (1990 pop. 723,959), coextensive with San Francisco co., W Calif., on the tip of a peninsula between the Pacific Ocean and San Francisco Bay, which are connected by the strait known as the Golden : Morgan Morgan, American family of financiers and philanthropists.

Junius Spencer Morgan, 1813–90, b. West Springfield, Mass., prospered at investment banking.

Licklider, J. C. R. (1965). Libraries of the future. Cambridge, MA: MIT Press.

Mackie-Mason, J. K.; Riveros, J. E; Bonn, M. S.; & Lougee, W. P. (1999). A Report on the PEAK experiment: Usage and economic behavior. D-Lib Magazine, 5(7/8). Retrieved September 29, 2000 from the World Wide Web: mackie-mason/07mackie-mason.html.

Marchionini, G., & Crane, G. (1994). Evaluating hypermedia hypermedia: see hypertext.

The use of hyperlinks, regular text, graphics, audio and video to provide an interactive, multimedia presentation. All the various elements are linked, enabling the user to move from one to another.
 and learning: Methods and results from the Perseus Project. ACM (Association for Computing Machinery, New York, A membership organization founded in 1947 dedicated to advancing the arts and sciences of information processing. In addition to awards and publications, ACM also maintains special interest groups (SIGs) in the computer field.  Transactions on Information Systems, 12(1), 5-34.

Marchionini, G.; Plaisant, C.; & Komlodi, A. (in press). The people in digital libraries: Multifaceted mul·ti·fac·et·ed  
Having many facets or aspects. See Synonyms at versatile.

Adj. 1. multifaceted - having many aspects; "a many-sided subject"; "a multifaceted undertaking"; "multifarious interests"; "the multifarious
 approaches to assessing needs and impact. In A. Bishop, B. Buttenfield, & N. VanHouse (Eds.), Digital library use: Social practice in design and evaluation. Cambridge, MA: MIT Press. Retrieved September 29, 2000 from the World Wide Web:

Okerson, A. (1999). Electronic collections development. Retrieved September 29, 2000 from the World Wide Web:

Rusbridge, C. (1998). Towards the hybrid library. D-Lib Magazine, 6(7/8). Retrieved September 29, 2000 from the World Wide Web: rusbridge/07rusbridge.html.

Saracevic, T., & Kantor, P. (1997). Studying the value of library and information services. I. Establishing a theoretical framework. II. Methodology and taxonomy taxonomy: see classification.

In biology, the classification of organisms into a hierarchy of groupings, from the general to the particular, that reflect evolutionary and usually morphological relationships: kingdom, phylum, class, order,
. Journal of the American Society for Information Science, 48(6), 527-542, 543-563.

Saracevic, T. (1999). Information science. Journal of the American Society for Information Science, 50(12), 1051-1063.

Shneiderman, B. (1998). Designing the user interface: Strategies for effective human-computer interaction (3d ed.). Reading, MA: Addison Addison, village (1990 pop. 32,058), Du Page co., NE Ill.; inc. 1884. An industrial suburb of Chicago, it manufactures machinery and plastic items.  Wesley Longman.

Su, L. T. (1992). Evaluation measures for interactive information retrieval. Information Processing & Management, 28(4), 503-516.

Wiseman, N.: Rusbridge, C.; & Griffin, S. M. (1999). The joint NSF/JISC International Digital Libraries Initiative. D-Lib Magazine, 5(6). Retrieved September 29, 2000 from the World Wide Web:

TEFKO SARACEVIC is a Professor at the School of Communication, Information, and Library Studies, Rutgers University Rutgers University, main campus at New Brunswick, N.J.; land-grant and state supported; coeducational except for Douglass College; chartered 1766 as Queen's College, opened 1771. Campuses and Facilities

Rutgers maintains three campuses.
. He teaches courses in various areas including digital libraries and principles of searching. Over the years, his research focused on the notion of relevance, human interaction in information retrieval and libraries, and the value of library and information services.

Tefko Saracevic, School of Communication, Information and Library Studies, Rutgers University, New Brunswick New Brunswick, province, Canada
New Brunswick, province (2001 pop. 729,498), 28,345 sq mi (73,433 sq km), including 519 sq mi (1,345 sq km) of water surface, E Canada.
, NJ 08901-1071
COPYRIGHT 2000 University of Illinois at Urbana-Champaign
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2000, Gale Group. All rights reserved. Gale Group is a Thomson Corporation Company.

 Reader Opinion




Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:evaluation criteria for design and management of digital libraries
Publication:Library Trends
Geographic Code:1USA
Date:Sep 22, 2000
Previous Article:Current Opportunities for the Effective Meta-Assessment of Online Reference Services.
Next Article:Project Ethnography: An Anthropological Approach to Assessing Digital Library Services.

Related Articles
The Impact of Electronic Resources on Serial Cancellations and Remote Storage Decisions in Academic Research Libraries.
Navigating the Parallel Universe: Education for Collection Management in the Electronic Age.
Evaluating Digital Libraries for Teaching and Learning in Undergraduate Education: A Case Study of the Alexandria Digital Earth ProtoType (ADEPT).
Digital Libraries and Their Challenges.
Evaluating Digital Libraries: A Longitudinal and Multifaceted View.
Current Opportunities for the Effective Meta-Assessment of Online Reference Services.
Project Ethnography: An Anthropological Approach to Assessing Digital Library Services.
The role of research in public libraries.
The Gulliver online database evaluation tool.

Terms of use | Copyright © 2014 Farlex, Inc. | Feedback | For webmasters