Current Opportunities for the Effective Meta-Assessment of Online Reference Services.

ABSTRACT

THE AUTHOR SUGGESTS THAT CURRENT CONDITIONS ARE nearly ideal for the effective meta-assessment of online reference services (ORS), in part because expected patterns and modes of service have not yet been established for emerging and evolving online environments, and in part because the distance between theory and practice regarding reference service in general is at its perigee. Meta-assessment is defined as the deliberate examination of the elements, basic conditions, and needs of a thing (service, event, system, and so on) that transcend particular instantiations of that thing. Meta-assessment does not assess particular programs but rather the conditions under which all online reference services must exist. Meta-assessment occupies the conceptual space between the philosophy of reference (i. e., the examination and articulation of first principles) and the assessment of a particular reference service program. The basic conditions for the evaluation and assessment of ORS are articulated and explored. The impact of "rogue" ORS (i.e., ORS that are not affiliated with any particular digital library) on the process of meta-assessment is examined. Several parameters of the basic conditions for reference service in any form and any environment are enumerated. Although the widespread recurring assessment of specific ORS may be a few months off, the window of opportunity for the effective meta-assessment of ORS in general may be closing.

INTRODUCTION

If environments are understood as places and spaces in which human beings pursue their wants and needs, technological developments are interesting because they change existing environments and create entirely new ones. The digital revolution has fostered many new environments that generally fall under the rubric of cyberspace. The digital libraries movement involves both substantial environmental change and the emergence of new service environments. New technologies at least encourage (and perhaps force) humans to re-examine what they think about environments, the horizons visible at the edges of known environments, and what new environments seem to offer and encourage.

Technological changes are affecting the environments and environmental conditions related to reference service. The emergence of digital libraries and other online environments force us to re-think the philosophy of reference. If assessment is defined as the deliberate process of comparing the reality of a service against both its stated goals and the realm of the possibilities for such a service, the process of assessment currently faces formidable challenges. It is extremely difficult to assess digital library public services, especially online reference services, in part because we do not really know yet what types of demanded services will emerge from users of digital libraries as they settle into these new online environments. We should not assume that simply transporting traditional real-world library services into the digital realm will be either necessary or sufficient.

One is reminded of the transformational processes of bringing water from the town well into individual homes. Simply delivering the same old water indoors seems at first to be a great leap forward--a triumph of the technological arm of civilization. As the first wave of enthusiasm for the home delivery of water subsides, however, homeowners begin to demand new services. They want their water to be heated, treated, filtered, and fluoridated. Although some of these new demands for services may have been predictable, some probably were not even imaginable to people who formerly trudged to the central village well to draw their water. When water was delivered into the home, a new service environment emerged. Similarly, just as the central library served as the font of information for an often geographically defined community of users (civic, educational, or corporate), when digital libraries began to deliver information directly into residences and workplaces, some of the demands for new DL services may have surprised and astounded both librarians and library users. For example, people will queue for services at the town well and the reference desk at the community library, but apparently they are unwilling to queue and wait patiently for similar services when delivered into domiciles, offices, and classrooms.

We should not assume that a digital library service must, by definition, be associated directly with a full-service digital library. It is possible to conceive of, and implement, a digital service (e.g., reference) without having it be part of a full-blown digital library. In online environments, information services typically associated with robust real-world libraries could float free from their tethered moorings. The economics of online information services may undergo a substantial alteration in the near future. Another reason online reference services may be difficult to assess is that "rogue" reference services have burst onto the scene (e.g., AskJeeves.com and WebHelp.com) that are not affiliated with any specific digital library. Suddenly the field of reference is flooded with a host of players who apparently are attempting to play the same game under various conditions and rules. How can we assess the value and worth of these rogue reference services? How will they affect the development of ORS within full-service digital libraries? Should we use the same criteria and standards that we would use for ORS that are affiliated with some sort of full-service DL? In the vast deserts of online environments, there probably is room for both types of ORS. Online reference service may be further subdivided into a wide variety of specialized services. Some of these specialized services may be amendable to commercial for-profit models while others may not.

The proliferation of rogue reference services raises an interesting meta-assessment question. What is the relationship between any reference service and the full library that surrounds it? What are the ties that bind a reference service to broader organizational goals? Some core real-world library functions, such as collection development, acquisitions, cataloging, and circulation do not make much sense without a collection. Other services (e.g., document delivery) rely on a collection, but it may not be a locally owned, housed, and controlled collection. Reference service seems to thrive when the human reference providers have ready access to some sort of extra-cranial collection of information, but it may not yet be self-evident what type of reference collection will be needed to optimally support an online reference service. Physical proximity is neither a necessary nor sufficient condition.

PHILOSOPHY, META-ASSESSMENT, AND ASSESSMENT

For these reasons, it may be difficult to assess current DL service programs (particularly ORS) in the context of the universe of all possible DL services. We simply have not yet discovered the entire online universe and the way humans will comport themselves in it, which in part is a process of self-discovery as one dives into a new pool of information. A philosophy of reference could be described as the examination and explication of the fundamental principles and goals of any and all reference services. Philosophy attempts to get at the essence of the thing, regardless of general and specific environmental conditions. The assessment of a specific reference service program examines how well a particular reference effort in a known environment is able to achieve its goals. Assessment certainly draws upon (and relies upon) a philosophy of reference, even if that philosophy is only implied in a pre-reflective fashion. At this moment in the overall development of reference services, a deliberate collaborative attempt at meta-assessment is warranted. Whereas an assessment project attempts to evaluate and assess a particular instantiated reference program, a meta-assessment project attempts to evaluate and assess the very preconditions, limitations, and assumptions upon which any online reference service relies. For the purposes of this article, meta-assessment is defined as the deliberate examination of the elements, basic conditions (necessary and sufficient), and needs of a thing (service, event, system, and so on) that transcend particular instantiations of that thing. A meta-assessment of ORS and surrounding online environments should be undertaken as a deliberate conceptual environmental scan and blueprint for future ORS.

Assessment and evaluation often are used almost interchangeably. Saracevic and Covi (2000) define evaluation as an appraisal of the performance or functioning of a system, or part thereof, in relation to articulated objectives. They insist that any evaluation must specify clearly what elements are being evaluated because an exhaustive evaluation of anything is impossible. Performance can be evaluated in terms of effectiveness, efficiency, or a combination of cost-effectiveness. This sense of evaluation could be understood as an "intra-system" definition, because the assessment or valuation of the system or service in relation to the broader environment, or in relation to all possible systems or services that could be designed and deployed to meet user needs, is not an integral part of this type of evaluation process. Questions such as "Why does this service have this set of articulated objectives rather than another set?" are difficult to address in an evaluation process so defined and pursued. For example, traditional real-world reference services tend to focus on helping users of the information system discover and retrieve information and information objects from the system. Helping the user with the myriad post-retrieval processing tasks is not a primary focus of traditional reference service, but it could be for DL reference services, in part because post-retrieval processing of digital information could become very complex and potentially valuable. At the early stages of DL design and deployment, we need an assessment and evaluation system that enables these kinds of meta-assessment questions to be raised and efficiently and satisfactorily addressed.

Meta-assessment occupies the middle ground between philosophy and assessment. Meta-assessment efforts must focus on the realm of the possible as well as on what actually has emerged as implemented services. Any assessment of a particular library program must be grounded somehow in both the philosophy and the meta-assessment of that aspect of librarianship. The meta-assessment of real-world reference services (centered around the reference desk concept) has not occupied the profession's attention for some time because the basic parameters of that environment are fairly well known. We know much more about human beings in real-world environments than about human beings in online environments.

ONLINE REFERENCE SERVICES

Reference services are one class of library public services. A simple definition of a library public service (in any environment--real or virtual) is any attempt by the library as a service organization to provide guidance and assistance to individual users of the library (usually via one-on-one, face-to-face, real-time, human-to-human interaction) as they search for, interpret, and gain value from both information objects and the ambient information environment. Any library public service that attempts to mediate between an articulated structured set of information and its users and potential users could be considered a public service.

Online reference services have been springing up all over cyberspace both inside full-service digital libraries and outside. For-profit and not-for-profit ORS have been successfully launched. Saracevic and Covi (2000) list three service constructs or elements within digital libraries: availability, range of available services, and assistance and referral. Janes, Carter, and Memmott (1999) define digital reference service as "a mechanism by which people can submit their questions and have them answered by a library staff member through some electronic means (e-mail, chat, Web forms, etc.), not in person or over the phone" (p. 146).

Saracevic and Covi (2000) note that the boundaries of any system often are difficult to determine, and ultimately the establishment of system boundaries involves rather arbitrary decisions. Every system and service exists within (and interacts with) a complex, comprehending environment, and the boundaries between the two are blurry. Meta-assessment activities explore the possible relationships between a system and its surrounding environment. Borgman (1999) observes that digital libraries are created by and for a community of users. The functional capabilities of a digital library support the information needs and uses of that community. On its Web site (www.clir.org/diglib/dlfhomepage.htm), the Digital Library Foundation defines digital libraries as "organizations that provide the resources, including the specialized staff, to select, structure, offer intellectual access to, interpret, distribute, preserve the integrity of, and ensure the persistence over time of collections of digital works so that they are readily and economically available for use by a defined community or set of communities" (Digital Library Federation, 1999). These last two definitions emphasize the community of users (and potential users) as a key environmental component of any DL service.

ASSESSMENT OF ONLINE REFERENCE SERVICES

Measuring use of a service may be inherently more complex than measuring the use of information content--or any type of tangible commodity, for that matter. Use of content, however, has its own set of measurement, interpretation, and assessment problems. Because the actual use of information is very difficult to isolate and measure, often use is measured in terms of allegedly good indicators of use. In real-world libraries, the circulation of the text-bearing device (often a book) serves as the primary measurable indicator of use. In the digital library arena, the display, downloading, and printing of files currently function as the primary indicators of use. High activity seems to indicate more use, which generally is perceived as good. When we turn our attention from content to services, however, the equation that worked well for the collection (where more use may serve as an indicator of a good program) is not as firm when the program is a public service, especially if the service purports to have some sort of educational component. If many users of a reference service are frequent, almost habitual, users of the service, this may be an indication that the reference service is not optimizing its impact as a form of education, not that the reference service is a runaway success. In addition, the accessibility of a service and the level of use appears to be directly related. If ORS are perceived as more accessible to an online population than traditional reference desk services are to a geographically circumscribed population, the raw usage of the ORS will be understandably higher.

Levy (2000) asserts that the lack of attention to preservation issues has been the most glaring omission of the first decade of digital library research and development. A strong argument could be made that lack of attention to DL public services is almost as glaring. The brief history of the development and deployment of digital libraries has an uncanny resemblance to the history of the development of real-world libraries. If public services are the third leg of the library stool (after collections and bibliographic records), evaluation and assessment projects and programs receive little attention until all three legs of the stool are firmly in place. Saracevic and Covi (2000) note that, during the explosion of DL funding and projects in the 1990s, evaluation was conspicuous by its absence in the vast majority of DL projects and programs, both research and practice oriented. Within the context of digital libraries, evaluation has not yet been specified as to what it might mean and entail. Saracevic and Covi raise the sobering question that it may be too early in the evolution of digital libraries to attempt to evaluate them in any formal way. In the booming frontier of DL development and deployment, no one has much interest in (and time for) formal evaluation and assessment activities. They assert that the conceptual nature of evaluation of digital libraries is too underdeveloped to be useful. Because the evaluation of DLs will be a complex undertaking, it needs to be understood both as a conceptual and a pragmatic challenge (Saracevic & Covi, 2000). Perhaps before we can assess ORS in any meaningful, sustainable, and generally useful way, we need to philosophize and "meta-assess" about the possibilities for library reference services in online environments. Such activities could substantially inform and influence the future development of digital library public services.

Because the current environment is changing rapidly and filled with unknown opportunities for digital library services, assessment tools, projects, and programs should focus on environmental conditions, including the needs and wishes of users of digital libraries, rather than on actually implemented projects and programs. We should evaluate digital library services in the context of all possible services rather than in the context of physical library services. We should not assume that the reason for a particular service is self-evident. As our attention shifts from a specific DL public service program toward an examination of all possible services within a new online environment, we are moving away from assessment toward meta-assessment.

Defining the assessment criteria for a DL public service is a matter of some debate. Buttenfield (1999) notes that patrons use digital libraries for tasks that are not feasible in traditional physical libraries, and sometimes they use the digital library in ways that are not anticipated by system designers. Although no one would argue that the reference librarian is the sole interface to the real-world library, in many ways the DL interface is the system. The interface is a gatekeeper to the collection and services. If the DL user cannot understand and navigate through the interface, the contents of that DL remain inaccessible (Buttenfield, 1999). "The dilemma is to evaluate a library which has not been completed and which is expected to become functionally more complex, using methods whose efficacy cannot be determined absolutely, and which may in fact not be appropriate" (Buttenfield, 1999, p. 43). Buttenfield sees numerous differences between evaluation procedures used for physical libraries and the procedures needed to evaluate digital libraries. Lankes and Kasowitz (1998, p. 180) suggest that digital reference services can learn from the research and implementation of evaluation programs of traditional library reference. More recently, Kasowitz, Bennett, and Lankes (2000) have written about quality standards for online reference service consortia designed initially for the Virtual Reference Desk AskA Consortium.

These are boom times for DL projects and the development of DL services. Lankes and Kasowitz (1998, p. 181) note that, because the concept of digital reference is so new, not much information is available that delineates best practices for the evaluation of digital reference and AskA services. Because digital libraries are both similar to, and different from, physical libraries, the challenge to digital library evaluation is to develop and apply new modeling and evaluation concepts and approaches (Saracevic & Covi, 2000). They suggest that evaluation criteria from previous evaluations of traditional libraries, information retrieval systems, and human-computer interactions could be adapted to evaluate digital libraries.

Saracevic and Covi (2000) identify at least two approaches to the evaluation of a digital library. The ethnographic approach "is highly appropriate for gaining a broad understanding of the role and effects of a practice or a construct in a wider social or organization framework." In a recent introductory essay to a special issue of the Journal of the American Society for Information Science on digital libraries, Hsinchun Chen (2000) noted that digital libraries are a type of information technology in which social impact matters as much as technological advancement, and that it is difficult to evaluate this type of new technology in the absence of real users and large collections. Chen seems to argue for the adoption of an ethnographic approach. Saracevic and Covi elect, however, to focus on the system approach because it is the most widely practiced or suggested approach for evaluating all kinds of information systems, including digital libraries.

Closely related to the ethnographic approach could be what we call the holistic approach. This approach tries to put the activities occurring within digital libraries in a broader (yet user-based) context. The approach involves asking questions such as: How well do the DL services help the user integrate information found in the DL with other information available to the user population? For example, the earth scientists who were a target user group for the Alexandria Digital Library (ADL) often work in sophisticated online information processing environments. They want and need the ADL to be tightly integrated with this overall online working environment in which they retrieve and manipulate data sets and digital maps (Hill et al., 2000, p. 250). The service goal has shifted from helping people identify and retrieve information from the library to interpreting and integrating the information into their present personal projects, whatever they may be.

POSSIBLE ROLES FOR ORS IN ONLINE ENVIRONMENTS

Any library service must mediate among the organization, policies, and procedures of the library and the needs and wants of the user population seeking library services. Providers of mediation services need to know the library, the users, and the basic conditions of the environment in which services are provided. In order to serve, a service unit must know its users. This is as true for rogue reference services as it is for public services nestled in a digital library. Most real-world library service units rely on self-declarations from users concerning their needs and wants. Digital libraries offer other ways to learn about users. ORS and associated software, much of it developed originally to serve as customer relationship management software for e-commerce sites, offers systematic ways to glean information from the reference interview.

What we need to assess are the several variable components of online reference services. A combination of controlled-environment and real-life research projects will be needed. Somehow we need to assess the possibilities of reference service in online environments in addition to the assessment of particular online reference pilot projects and programs. Meta-assessment is proposed as this type of higher level assessment of environmental conditions and opportunities, including new and evolving end-user needs and wants.

The role of online reference services in online environments remains an open question. What roles do public services play in any library organization, regardless of the virtual or real spaces and service areas of the library? Are public services for emerging digital libraries even necessary? We may be witnessing (in an accelerated fashion) the prototypical stages in the development of libraries. Libraries began with collections of documents--usually books in libraries that developed during and after the Enlightenment. Then came metadata records and databases about the items in the collection and the organization among them. In the good old days, organized collections of metadata records were known as accession lists, finding aids, and catalogs. The third stage in the development of libraries is the emergence of public services. Reference services, library instruction, reader's advisory services, research services, and other public services all emerged quite some time after the existence of collections and metadata about collections. Although this is all recent compressed history, with the DL movement, the collections generally arrived first, followed by a concern for metadata and other aspects of organization and intellectual discovery, followed at last by a concern for public services in online environments.

Marchionini (1999) observes that, even with powerful search tools, people often are unable to solve their information problems on their own. He notes that the need for reference librarian assistance remains a significant challenge in digital libraries. People become lost more easily in digital libraries. Perhaps an online service program is needed that actively seeks to find or detect lost information seekers. It may not be in the best interests of DL users to wait for them to become aware and declare that they are in need of DL services. The onus of service initiation should be on the system, not on the user.

Online reference services also are dealing with communities that organize themselves differently (and perhaps seek information differently) than do traditional user communities. Collaborative seeking and use of information may be a prominent feature of the future landscape, and digital library services will need to be designed to facilitate group and team activity, not solitary information hunters. Marchionini (1999) advocates a model of digital library development that would augment existing real-world library services by facilitating community-based sharing of time and information. He calls this new type of digital library, where people and their interactions are as important as the structured environment of information objects, a "sharium." A sharium would be a new type of DL that combines elements of learning communities, scientific collaboratories, and special libraries to facilitate communication and to distribute the load of solving information problems among a group of people (Marchionini, 1999). The services within the sharium would facilitate group problem solving and intellectual exchanges. The real-world library model, where asynchronous communication is encouraged but synchronous communication is discouraged (e.g., through prohibitions against talking and chat room discussions), would be augmented by the sharium service model where synchronous communication is fostered and facilitated, within established norms of acceptable social behavior, of course. The sharium concept involves a more broad-based, intra-communal sharing of expertise than is commonly found in traditional reference services.

In addition to services that will help individual users (or small work groups) customize an existing DL environment to meet their particular needs, digital libraries also could provide services that facilitate end-user input into the basal structure and content of the DL environment. Real-world libraries and library services were not designed to systematically and continuously seek input from the end-user population. For example, despite heavy use of OPACs over several decades, what traces of influence from actual users are manifest in these systems as they exist today? Online booksellers, such as Amazon.com, have learned that users want to talk about and assess their use of information objects. In plain English, readers want to recommend or warn other potential readers about good and bad books. Amazon.com provides a carefully designed service to provide an outlet for this basic human need. The exploration and use of an information space creates expertise. A smart information space would contain services that help tap that expertise to assist other users and to improve the information space itself. In Marchionini's conception, the sharium, as an information-rich collaborative environment, would encourage the user community itself to share expertise and time to add new value to digital libraries.

META-ASSESSMENT OF ONLINE REFERENCE SERVICES

We currently enjoy a unique historical opportunity. As reference service migrates to online environments (probably without abandoning real-world environments), we have the opportunity to examine the fundamental assumptions and foundations of reference service in a changing environment wherein our examinations really could make a difference in what actually evolves. The meta-assessment of online reference services may be uniquely effective at this time in history because the distance between theory and practice is at its perigee. By arguing for a historical window of opportunity where meta-assessment may materially affect the reality of online reference services, the author is not suggesting that at most other times the meta-assessment of reference services would be useless. In assessing reference service in online environments at this moment in history, we need to raise several high-level meta-assessment questions. It is not too late. As Janes, Carter, and Memmott (1999) note, although the Internet has been broadly available for years, the use of the Internet as a medium for the reference process in academic libraries is still in its infancy. We must not dawdle, however. Although some theoreticians and practitioners may argue that it is too early to meaningfully assess operational online reference services, it may be almost too late to undertake a meta-assessment of ORS in general, if one anticipated outcome of such an assessment is to significantly influence their design, development, and deployment.

A meta-assessment undertaking could focus on raising several key questions and assumptions about reference service in general and online reference services in particular. Once the questions have been raised and the assumptions questioned, they can be examined and tested--in thought experiments, in controlled research environments, and in pilot programs.

* To be successful, does online reference service have to be conducted in real time?

Examples of delayed service include snail-mail reference, delayed phone reference (where the user calls in the question, then the service provider phones back at a later time with the answer or assistance), and e-mail reference. None of these forms of reference service have proven to be widely accepted and frequently useful to a service population. Although all reference service involves some sort of time delay, it appears to be true that, for most users and most reference needs, delays of more than a few minutes significantly diminish both the usefulness and use of a reference service that routinely incorporates such delays into its service architecture. The challenge for designers of ORS becomes how to provide real-time reference service in a 24/7 online environment. Customer relationship management software appears to present some currently available, relatively low bandwidth, options for real-time online reference services.

* Should online reference service involve nonverbal communication?

Reference programs that attempt to circumvent both real-time service and the use of nonverbal communication may be seriously structurally disadvantaged. Based on a random survey of academic library Web sites conducted in May 1999, Janes, Carter, and Memmott (1999) found that none of the surveyed libraries used other means (beyond e-mail and Web forms) of answering digital reference questions, such as real-time chat, ICQ, and video technologies.

* Should ORS consist primarily of human-to-human communication?

The fundamental question concerning digital library public services is: What is the value of human intervention? Do humans need to search for information, and do other humans need to help those humans search for (and interpret) information, or can machines (hardware and software) assume increasing responsibility for meeting these human information needs? The role of humans in the emerging global information infrastructure is not obvious and assured. Frequently asked question (FAQ) documents or in-house collections of previously asked questions are common alternatives to human-to-human communication. Perhaps another way to state this fundamental question concerning reference service: Does each reference interaction need to be treated as a unique information need or can canned responses to previously asked, researched, and answered reference queries be designed into a reference service in a manner

that is truly acceptable and useful to the users of the service? We should not assume that DL services must be based on human-human interaction. This type of interaction may be the most expensive, but it also may be the most complex and fruitful. A DL service program could be evaluated and assessed based on its ability to maximize the fruitfulness of the human-human interaction while minimizing the expense. For good successful online reference services, human-human interaction may be a luxury, not the typical modus operandi.

* How should knowledge and navigational expertise within a given population be gleaned from and diffused throughout that population?

One goal of any reference service is to provide an effective, efficient, and accessible mechanism for identifying (or fostering the development of) expertise, then diffusing that expertise among the population of willing and needful recipients of that expertise. Currently there is much discussion and hope that the processes of seeking and using information in online environments will be much more collaborative than in real-world libraries. If collaboration becomes all the rage among the DL user population, the impact on DL service programs could be profound. The question becomes: How can a system extract and efficiently distribute expertise in a collaborative environment? The service paradigms that made sense in real-world libraries where scholarship and learning often were solitary pursuits may need to be rethought and revised for the emerging collaborative DL environments. If reference assistance is broadly conceived, peer assistance may gain market share at the expense of expert assistance.

The challenge of how to extract expertise from a given population, then distribute it in a meaningful way throughout a given population, is both daunting and exhilarating. Perhaps at one extreme is traditional reference desk service, where a reference librarian (or small group of librarians) serves as a conduit for theoretically all knowledge and structural/navigational expertise through an information space. Perhaps at the other extreme is a community of users defined by a listserv discussion list. If a user needs reference assistance, he or she posts the question or problem to the list, then lets individuals from the community self-select to attempt to meet the information need. The questioner may receive no responses to the query or many responses of varying quality. Between these two extremes of reference service are many potential service models.

Ultimately, any reference service in any environment is a system for finding needed expertise within a system and disseminating that knowledge and expertise to other areas of the defined environment that are in need of that expertise. In the traditional model of providing reference service via a physical reference desk, a single reference librarian often draws on his or her training, own knowledge (of topics and the structure of information sources), and various information sources to meet the articulated information and expertise needs of the reference seeker. The reference interaction is the elaborate process by which this need for expertise is communicated and (it is hoped) satisfied. This traditional mode of expertise mining and dissemination has proven to be quite successful. As reference service moves into online environments, however, we are compelled to ask the fundamental question again: What is the most efficient way to find, translate, and transmit expertise in online environments? In the long run, we may discover that our early attempt to morph the traditional mode of real-world reference service to meet our unfolding comprehension of what online environments offer and demand as environments is not a particularly efficient and effective way to pursue the enduring need for reference service in online environments.

* Is software a crucial facet of an online reference service assessment project?

Because software (including human use of software) really defines the environment in which online reference services come into existence, the functionality and performance of the supporting software is crucial. We need methods for understanding and reviewing software that go beyond (or perhaps delve more deeply into) the look and feel and basic functionality of the software. This sophisticated software is capable of creating complex online environments and interactions. There seems to be nothing analogous to this facet for the assessment of real-world reference service other than perhaps assessing the layout of the reference desk and the ready reference collection--a rather simple assessment challenge compared to the task we face with assessing the supporting software.

META-ASSESSMENT OF ASSESSMENT METHODS

Every act of assessment should be at least a little self-reflective. Buttenfield (1999) argues that a good DL assessment program should evaluate not only the DL itself but also the evaluation methods. Such a self-referring analysis, termed a double-loop paradigm, allows evaluators to identify the relative efficacy and efficiency of particular assessment methods for specific situations. In addition to assessing the value of what is being assessed, the assessment activity should focus on the context and value of the assessment activity itself. Meta-assessment efforts also can examine the basic methods of an assessment program. For example, computerized monitoring (i.e., the use of computers to monitor human-computer interaction) in general, and transaction log analysis in particular, are interesting in the context of evaluating digital library services because they represent the possibility of making a service or online information environment automatically customized or self-improving. The system can contain within its programming the seeds of a self-evaluation. (The "self" here is the human-computer interaction as a series of events, or the online environment itself, not the human self.) A self-improving environment utilizes actual use of the system or service as data for making decisions and design changes that ostensibly improve the service for present and future users, both collectively and individually. Buttenfield (1999) states: "Ideally, one would prefer a self-evaluating and self-modifying system ..." (p. 54). Hill et al. (2000) recommend that session logs be available to ADL (Alexandria Digital Library) help desk personnel as they interact with users (p. 257). In mid 1998, the Alexandria Digital Library was using a registration form with controlled response options, session IDs linked with user IDs, session logs, and exit polls for user comments and evaluations after each session to obtain user feedback (Hill et al., 2000, p. 248). Buttenfield (1999) describes how the evaluation of the Alexandria Digital Library project used the convergent methods of transaction log analysis, talk-aloud protocols, and exit interviews to learn more about user behavior and motivation.

CONCLUSION

Libraries can be understood as a communication medium. Content creators (the artists formerly known as authors) and content users communicate over space and time. The emergence of any new communication medium creates needs for new services. The telephone led to directory assistance, maps of area codes, white pages, and yellow pages. The television created the need for TV Guide, new types of furniture (e.g., La-Z Boy loungers and home entertainment centers), and microwave popcorn. It seems almost certain that the emergence and acceptance of digital libraries will lead to the creation of new services that meet the needs of online users. Needs assessment, a form of meta-assessment, should precede the creation and assessment of programs. Our professional challenge is to discern these new needs, then meet them. Discerning nascent and emerging needs for a service program is a form of assessment. This type of assessment could serve as a bridge between pure meta-assessment and more

traditional assessment activities.

Currently there is a strong division between real-world (physical) and online environments. As computing becomes more diffused throughout the real-world environment and throughout human experience within real-world environments, the current cognitive disconnects between being online and being in the real world may pass away. Eventually, real-world library services and digital library services may meld into a seamless whole.

In real-world libraries, at any given time physical objects can be organized only in one way. From all of the possible ways of organizing the physical objects, one had to be chosen, and the choice was made by someone other than the user prior to the moment of exploration and use. The service programs of real-world libraries often focus (rightly so) on explaining these "a priori" organizational choices to the user population. For example, the simple statement from a reference librarian that "unbound journal issues are located in the current periodicals room on the second floor" contains a wealth of clues to the user about how the real-world library has been chosen to be organized and how materials are processed within that library. The digital library, however, does not operate under this fundamental constraint of real objects. It can be organized in multiple ways and, perhaps more importantly, the user can control (either wittingly or unwittingly) how the DL is organized and presented. Suddenly, the service program of the DL appears to be built on quicksand.

Saracevic and Covi (2000) observe: "The evaluation of digital libraries is still in a formative stage. Concepts have to be clarified first. This is the fundamental challenge for digital library evaluation." Meta-assessment is one way to clarify some basic concepts of online reference services. Although the widespread recurring assessment of specific online reference service programs may be a few months off, the window of opportunity for the optimally effective meta-assessment of ORS in general may be closing.

REFERENCES

Borgman, C. L. (1999). What are digital libraries? Competing visions. Information Processing & Management, 35(3), 227-243.

Buttenfield, B. (1999). Usability evaluation of digital libraries. Science and Technology Libraries, 17(3/4), 39-59.

Chen, H. (2000). Introduction to the special topic issue: Part 1. Journal of the American Society for Information Science, 51(3), 213-215.

Digital Library Federation. (1999, April 21). A working definition of the digital library. Retrieved October 3, 2000 from the World Wide Web: http://www.clir.org/diglib/ dldefinition.htm.

Hill, L. L.; Carver, L.; Larsgaard, M.; Dolin, R.; Smith, T. R.; Frew, J.; & Rae, M-A. (2000). Alexandria Digital Library: User evaluation studies and system design. Journal of the American Society for Information Science, 51(3), 246-259.

Janes, J.; Carter, D.; & Memmott, P. (1999). Digital reference services in academic libraries. Reference & User Services Quarterly, 39(2), 145-150.

Kasowitz, A.; Bennett, B.; & Lankes, R. D. (2000). Quality standards for digital reference consortia. Reference & User Services Quarterly, 39(4), 355-363.

Lankes, R. D., & Kasowitz, A. S. (1998). The Ask A Starter Kit: How to build and maintain digital reference services. Syracuse, NY: ERIC Clearinghouse on Information & Technology, Syracuse University (ERIC Document Reproduction Service No. ED 427 779).

Levy, D. M. (2000). Digital libraries and the problem of purpose. D-Lib Magazine, 6(1). Retrieved October 3, 2000 from the World Wide Web: http://www.dlib.org/dlib/ january00/01levy.html.

Marchionini, G. (1999). Augmenting library services: Toward the sharium. Retrieved October 3, 2000 from the World Wide Web: http://ils.unc.edu/~march/sharium/ISDL.pdf.

Saracevic, T., & Covi, L. (2000). Challenges for digital library evaluation. In Proceedings of the American Society for Information Science Annual Meeting (Chicago, IL, November 11-16, 2000).

THOMAS A. PETERS is the Director of the Center for Library Initiatives at the Committee on Institutional Cooperation (CIC), the academic consortium of the Big Ten Universities and the University of Chicago. An academic librarian for thirteen years, Mr. Peters has worked previously at Western Illinois University in Macomb, Northern Illinois University in Dekalb, Minnesota State University at Mankato, and the University of Missouri at Kansas City. He earned his library science degree at the University of Iowa. His second master's in English was completed at the University of Missouri at Kansas City. He has written two books, the most recent being Computerized Monitoring and Online Privacy (1999, McFarland).

Thomas A. Peters, Center for Library Initiatives, Committee on Institutional Cooperation, 302 East John Street, Suite 1705, University of Illinois, Champaign, IL 61820-5698