Printer Friendly
The Free Library
22,725,466 articles and books

Outcomes assessment in the networked environment: research questions, issues, considerations, and moving forward.


THIS ARTICLE IDENTIFIES A NUMBER OF RESEARCH TOPICS related broadly to outcomes assessment in a networked environment and discusses issues affecting these research topics. It also proposes a framework to relate traditional evaluation components and terminology to the networked environment and identifies a number of factors in the networked environment that affect outcomes and other assessment methods. The article suggests that outcomes assessment has the potential to complement other assessment techniques to better assist libraries and related information organizations enhance their decisions in the provision of information services See Information Systems.  and resources. Given the increased rate at which libraries are using the networked environment to provide services and resources, however, much work remains before most libraries can implement outcomes assessment efforts successfully.


Organizations of many types in various operating environments In computing, an operating environment is the environment in which users run programs, whether in a command line interface, such as in MS-DOS or the Unix shell, or in a graphical user interface, such as in the Macintosh operating system.  have used a variety of performance assessment activities for some time. Libraries are no different, having engaged in the use of output, service quality, performance indicator, balanced scorecard Balanced Scorecard

A performance metric used in strategic management to identify and improve various internal functions and their resulting external outcomes. The balanced scorecard attempts to measure and provide feedback to organizations in order to assist in implementing
, and a number of other performance measurement techniques. At the core of these measurement activities are a number of basic questions:

* What resources are required to support the services that a library provides?

* What services and/or and/or  
Used to indicate that either or both of the items connected by it are involved.

Usage Note: And/or is widely used in legal and business writing.
 resources is a library able to provide with its investments in library infrastructure (broadly defined as personnel, technology, collections, facilities, etc.)?

* Are the library's customers receiving value out of the community's investment in library services?

* What is the real and/or perceived per·ceive  
tr.v. per·ceived, per·ceiv·ing, per·ceives
1. To become aware of directly through any of the senses, especially sight or hearing.

2. To achieve understanding of; apprehend.
 quality of library services?

* What are the impacts of library services and resources on the community that the library serves?

* What level of effort is required by library staff to implement measurement activities?

* In what ways does the library contribute to the overall mission, goals, and objectives of the community--or institutions within that community--the library serves?

The role and usefulness of outcome measures and outcomes assessment in answering these and related questions, although potentially rich, requires significant additional review, research, field testing, and development.

The purpose of this article is to:

* Identify selected research questions regarding outcomes assessment;

* Review a number of issues affecting outcomes assessment in general and outcomes assessment in a networked environment in particular; and

* Propose a framework to relate and better understand traditional evaluation components and terminology.

The article suggests that outcomes assessment has the potential to evolve to a point at which it will complement other assessment techniques that assist libraries and related information organizations make better decisions in the provision of information services and resources. There are, however, a number of research, methodological, and other issues that require consideration prior to reaching this potential. Given the library community's increased reliance on providing services and resources via the networked environment, much work lies ahead before libraries can engage successfully in outcomes assessment activities.


Outcomes assessment is a relatively new activity to libraries, and as such, this is a very broad area for research. Indeed, as with any new area of scholarly endeavor, there are many more questions than answers. Some key research questions are:

* What are the prevailing models/approaches for library services/resources assessment?

* Are these models distinct or interrelated in·ter·re·late  
tr. & intr.v. in·ter·re·lat·ed, in·ter·re·lat·ing, in·ter·re·lates
To place in or come into mutual relationship.


* To what degree do these models/approaches consider networked services/resources?

* How are "outcomes" considered in these various models?

* What do these models/approaches tell researchers and practitioners about use, uses, quality, impact, etc. of library services/resources?

* What outcomes assessment methods will be most useful and successful given the needs of a library, the resources available to the library for assessment, the assessment activity objectives, staff skills available, and other situational factors?

* What are the key variables to consider in the development of such models?

This article provides a foundation upon which to begin addressing these questions. The authors do not claim, however, that these are the only research questions that require attention regarding library outcomes assessment activities. These questions are, though, essential to further the development of outcomes assessment approaches that provide libraries with techniques that identify the impacts and benefits of library services/resources on the communities that they serve. The remainder of the article presents the issues and considerations regarding these research questions as a means of moving the research agenda forward.

The Current Context and the Need for Answers

The environment in which libraries find themselves at present creates a situation in which answers to the above posed research questions are essential. Libraries of all types, but public and academic libraries in particular, find that they are considered increasingly as part of the larger organizational structure This article has no lead section.

To comply with Wikipedia's lead section guidelines, one should be written.
 in which they reside--university departments or local government agencies. As such, library funding and continued well-being is no longer distinct from that of any other campus or local agency. Because of this situation, libraries are being asked to:

* Articulate articulate /ar·tic·u·late/ (ahr-tik´u-lat)
1. to pronounce clearly and distinctly.

2. to make speech sounds by manipulation of the vocal organs.

3. to express in coherent verbal form.

 the importance of and need for their services and resources;

* Identify the use and uses of their services and resources; and

* Establish the value, impacts, and benefits that the community receives from the library services and resources.

Given the above circumstances CIRCUMSTANCES, evidence. The particulars which accompany a fact.
     2. The facts proved are either possible or impossible, ordinary and probable, or extraordinary and improbable, recent or ancient; they may have happened near us, or afar off; they are public or
, it behooves the library practitioner and research communities to have assessment tools and approaches that enable libraries to articulate their contribution to the well-being of the communities that they serve.

In addition to this management context, there is the evolving technology context in which libraries operate. To state the obvious, library services and resources rely increasingly on:

* Technologies that continue to change and evolve at an ever-intensifying pace. On the one hand, these technology changes enable new services and resources that allow libraries to better meet the service and resource needs of their customers. On the other hand, this continual cycle of adoption, change, and new and/or enhanced services Enhanced service is service offered over commercial carrier transmission facilities used in interstate communications, that employs computer processing applications that act on the format, content, code, protocol, or similar aspects of the subscriber's transmitted information;  and resources creates a number of challenges--including assessment challenges--for libraries.

* Leased and network-based resources and situations that remain outside the direct control of libraries (see Figure 1). For example, libraries continue to grow their subscriptions to online databases and resources (e.g., e-books). Moreover, customer access to those services and resources can occur through a number of nonlibrary venues--each of which has implications for what libraries and vendors can collect and report regarding those user-initiated activities. As such, libraries are in a situation in which they do not control the services themselves (libraries are merely subscribers to resources such as EbscoHost, ScienceDirect ScienceDirect is one of the largest online collections of published scientific research in the world. Produced by Elsevier it contains over 8.5 million articles from over 2000 journals, including titles such as The Lancet, Cell and Tetrahedron , and NetLibrary), do not control the path through which customers gain access to the service/resource (remote access is beyond the scope of the library), and do not control the customer service/resource use and interaction data. All of this has a direct impact on the assessment activities in which a library can engage, the findings derived from such activities, and the types of questions that libraries can ask users regarding their use and/or assessment of the services/resources accessed (Shim A small piece of software that is added to an existing system program or protocol in order to provide some enhancement.

(jargon, memory management) shim - A small piece of data inserted in order to achieve a desired memory alignment or other addressing property.
 & McClure Mc·Clure   , Samuel Sidney 1857-1949.

Irish-born American editor and publisher who founded McClure's Magazine (1893), an influential muckraking periodical.
, 2002).


Added to this mix is the impression that some outside the library community regard the information content as available without charge to Internet users Internet user ninternauta m/f

Internet user Internet ninternaute m/f 
. As libraries struggle to identify, define, and articulate the impacts derived by customers through the use of library services and resources, libraries must educate the communities that they serve regarding the issues associated with networked-based information content.

There are a number of evaluation methodologies that exist--for example, return on investment, quality assessment, outcomes assessment, outputs, best practices--to assist libraries in identifying and communicating their service/resource impacts on their communities. There are also a number of situational factors that affect a library's or a researcher's ability to identify, study, and present results of that work. This article describes a selected number of those methodologies, explores the relationship between them, and presents issues associated with methods and what they can tell us about library services and resources.


A first step in addressing the research questions posed earlier is defining and understanding an "outcome." There is no single concise definition of what an "outcome" means in the context of library service. During the past decade, a number of writers and researchers have proposed definitions that Table 1 summarizes. A review of various definitions, however, does yield a number of common elements. In general, outcomes:

* Include the notion of an impact, benefit, difference, or change in a user, group, or institution based on the use of or involvement with a library service or resource;

* Are predetermined pre·de·ter·mine  
v. pre·de·ter·mined, pre·de·ter·min·ing, pre·de·ter·mines
1. To determine, decide, or establish in advance:
 based on a service/resource planning process in which the library engages to produce desired service/resource outcomes through the setting of service/resource goals and objectives; and

* Involve measuring and demonstrating the extent to which library services/resources meet the anticipated outcomes determined by the library or imposed by the community the library serves (e.g., academic institution, county, city).

These definitions broadly, therefore, consider outcomes assessment to be a proactive endeavor on the part of the library in which there is an a priori a priori

In epistemology, knowledge that is independent of all particular experiences, as opposed to a posteriori (or empirical) knowledge, which derives from experience.
 determination of the library service/resource preferred outcomes.

In general, these definitions assume that there is a larger context in which libraries reside that provides the basis for the library service/resource outcomes. In the case of academic libraries, for example, the desired outcomes might take the form of accreditation accreditation,
n a process of formal recognition of a school or institution attesting to the required ability and performance in an area of education, training, or practice.
 standards set forth by accreditation bodies; accountability The traceability of actions performed on a system to a specific system entity (user, process, device). For example, the use of unique user identification and authentication supports accountability; the use of shared user IDs and passwords destroys accountability.  measures imposed by university administrators/ boards; or state government-imposed higher education higher education

Study beyond the level of secondary education. Institutions of higher education include not only colleges and universities but also professional schools in such fields as law, theology, medicine, business, music, and art.
 outcomes. Finally, these definitions assume that libraries are able to measure their service/resource outcomes with reliable and valid instruments and demonstrate subsequently that library service/resource outcomes contribute to the mission, goals, and objectives of the community that the library serves.

Over time, library researchers, managers, and others have developed a number of models to describe and relate inputs, outputs, service quality, outcomes, impacts, etc. Kyrillidou (2002) summarizes some of these approaches. There is no lack of proposed models to describe these evaluation components (e.g., Hernon, 2002, p. 55; Cook & Heath heath, tract of open land
heath, tract of open land characterized by a few scattered trees, abundant moss cover, and numerous low shrubs, principally of the heath family (see heath, in botany).
, 2001, p. 580; Dresang & Gross, 2001, p. 28). While some might suggest that the plethora plethora /pleth·o·ra/ (pleth´ah-rah)
1. an excess of blood.

2. by extension, a red florid complexion.pletho´ric

 of views, models, and definitions describe a healthy intellectual development in library/information services and resources evaluation, others--namely many practitioners--may not agree. Part of the issue is that each of these views, models, and evaluation approaches Evaluation approaches are conceptually distinct ways of thinking about, designing and conducting evaluation efforts. Many of the evaluation approaches in use today make truly unique contributions to solving important problems, while others refine existing approaches in some way.  is presented individually without any review or consideration of the relationship between the key aspects of these models or perspectives.

Outcomes assessment, although not new to other organizations and sectors, is relatively new to libraries and focuses on determining the impact of a library's services and/or resources on its customers. In the broadest sense, outcomes assessment focuses on the extent to which a library's services and/or resources made a difference in the life of the library's individual, group, or institutional users (see Table 2). Writers and researchers, however, do not agree on what it means to measure outcomes.

Kyrillidou (2002) presents three models that depict de·pict  
tr.v. de·pict·ed, de·pict·ing, de·picts
1. To represent in a picture or sculpture.

2. To represent in words; describe. See Synonyms at represent.
 the possible relationship between inputs, outputs, outcomes, and service quality. These models include a linear model, cyclic model In the 1930s, theoretical physicists, most notably Einstein, considered the possibility of a cyclic model for the universe as an (everlasting) alternative to the Big Bang. However, work by Richard C. , and spiral spiral /spi·ral/ (spi´ral)
1. helical; winding like the thread of a screw.

2. helix; a winding structure.
 swirl (p. 44). Without empirical testing, Kyrillidou finds problems with all the models presented save the spiral swirl as it apparently can depict motion and a more flexible and dynamic process that intertwines inputs, outputs, outcomes, and service quality. Ultimately, Kyrillidou concludes that service quality is an outcome of library services and resources that exist through a library resource input and output process.

Hernon (2002) reaches a different conclusion, stating that service quality and satisfaction and outcomes assessment "truly stand out as central assessment concepts for librarianship li·brar·i·an  
1. A person who is a specialist in library work.

2. A person who is responsible for a collection of specialized or technical information or materials, such as musical scores or computer documentation.
. Inputs and outputs are important to institutions and accrediting bodies as measure of efficiency and crude measure of effectiveness" (p. 55). In Hernon's view, therefore, there is little to no relationship between inputs, outputs, service quality, and outcomes assessment--and libraries should focus on service quality and outcomes assessment techniques. King et al. (2002) take a different approach by linking types of measurement perspectives (library, user, organization, etc.) with specific types of measures--one of which is outcomes. The approaches shown in Table 2 are illustrative il·lus·tra·tive  
Acting or serving as an illustration.

il·lustra·tive·ly adv.

Adj. 1.
 of recent writings, though historically many writers have offered models and approaches related to inputs, measures, service quality, and outcomes.

Table 2, based on research by the authors and others, summarizes the view of the outcomes assessment environment of public and academic libraries from selected perspectives. Each of the described approaches has strengths and weaknesses; each makes implicit assumptions about outcomes and the relationships between outcomes and other evaluation factors; and most do not address a range of factors and considerations discussed later in this article. In short, it is not possible to address meaningfully the research questions posed initially in this article with such different views of outcomes and the relationship of outcomes to other evaluation approaches.

The authors of this article present a view summarized in Figure 2. This model, based on a number of large-scale large-scale
1. Large in scope or extent.

2. Drawn or made large to show detail.


1. wide-ranging or extensive

 field-tested and empirical research Noun 1. empirical research - an empirical search for knowledge
inquiry, research, enquiry - a search for knowledge; "their pottery deserves more research than it has received"
 efforts, posits that inputs and outputs form the basis for service quality and outcomes assessment activities (Bertot, McClure, & Davis, 2002; Shim et al., 2002; Bertot, McClure, & Ryan Ryan may refer to: Places
  • Division of Ryan, an electoral district in the Australian House of Representatives, in Queensland
  • Ryan, Iowa
  • Ryan, Oklahoma
  • Ryan Township, Pennsylvania
  • Ryan, New South Wales
Film and television
, 2001). In this view, inputs are the resources that libraries invest (e.g., money, staff, workstations, online commercial databases) in order to produce outputs (e.g., number of users of the workstations, number of database content downloads, circulation of material). Quality assessment involves determining the degree to which users find the library services/resources (outputs) to be satisfactory. Outcomes assessment, however, seeks to determine the impact of the library's services/resources (again, outputs) on the library service and resource users. In other words Adv. 1. in other words - otherwise stated; "in other words, we are broke"
put differently
, one cannot have outcomes measures without measures of outputs--libraries need to know what investments (inputs) produce what services (outputs) in order to determine the perceived quality (quality assessment) and impacts (outcomes) of those services/resources.


An Outcomes Framework--More than One Type of Outcome

Much of the outcomes literature presents outcomes as monolithic--that is, there is a set of generic outcomes that fit various organizational settings (IMLS IMLS Institute of Museum and Library Services
IMLS Institute for Museum and Library Services (US)
IMLS Institute of Medical Laboratory Sciences
, 2000). However, it is important to recognize that there are many types of outcomes, such as research and learning outcomes (Hernon & Dugan |

Dugan is a name of Irish heritage. Other common modern variants are Duggan, Dougan, Doughan, Doogan, and Duggin.
  • Coat of Arms: azure, a crescent argent between nine stars of eight points
  • Motto: Virtve et Valore
, 2002) as well as institutional outcomes (Fraser Fraser, river, Canada
Fraser, chief river of British Columbia, Canada, c.850 mi (1,370 km) long. It rises in the Rocky Mts., at Yellowhead Pass, near the British Columbia–Alta. line and flows northwest through the Rocky Mt.
 & McClure, 2002). Developing a framework for outcomes assessment requires a complex analysis that encompasses the operating environment of the library, the impact of situational factors on library services and resources outcomes, and the reality that it is not always possible for libraries to anticipate and/or predict the outcomes of their services/resources on users.

Different types of outcomes have been suggested by Lance et al. (2002), Hernon and Dugan (2002), Bertot (2001), and Fraser and McClure (2002)--to name a few. As a result, developing a framework for outcomes assessment requires a complex analysis that considers a number of outcome types:

* Economic. Economic outcomes would include the impact of library services and resources on the ability of library users to prosper financially, seek employment successfully, or develop and sustain a small business in the community;

* Learning. Learning outcomes would include the impact of library services and resources on the ability of library users to engage in lifelong learning Lifelong learning is the concept that "It's never too soon or too late for learning", a philosophy that has taken root in a whole host of different organisations. Lifelong learning is attitudinal; that one can and should be open to new ideas, decisions, skills or behaviors. , interact with and engage a number of information resources (1) The data and information assets of an organization, department or unit. See data administration.

(2) Another name for the Information Systems (IS) or Information Technology (IT) department. See IT.
, develop information literacy Several conceptions and definitions of information literacy have become prevalent. For example, one conception defines information literacy in terms of a set of competencies that an informed citizen of an information society ought to possess to participate intelligently and  skills, develop technology skills, become literate, or develop an analytic an·a·lyt·ic or an·a·lyt·i·cal
1. Of or relating to analysis or analytics.

2. Expert in or using analysis, especially one who thinks in a logical manner.

3. Psychoanalytic.
 ability to assess the validity and reliability of information sources;

* Research. Such outcomes might include the impacts of library services and resources on the research process of faculty and students in a university--e.g., assistance in proposal writing, grant receipt, and publication;

* Information Exchange. Information exchange outcomes would include the impact of library services and resources on the ability of library users to interact with government, exchange information with distant family, or receive information regarding countries of interest (e.g., foreign newspapers and other sources);

* Cultural. Cultural outcomes would include the impact of library services and resources on the ability of library users to develop an appreciation for fine arts, history, music, diversity, or other societal so·ci·e·tal  
Of or relating to the structure, organization, or functioning of society.

so·cie·tal·ly adv.

 aspects; and

* Community. Some library outcomes affect the local community, be that an academic setting, a city or town, or a virtual community. Such outcomes could affect the overall quality of life for members of the community, attitudes of community members toward services, or even the political landscape of the community.

These categories are not mutually exclusive Adj. 1. mutually exclusive - unable to be both true at the same time

incompatible - not compatible; "incompatible personalities"; "incompatible colors"
 but rather are intended to be illustrative of types of outcomes. It is possible to identify other types of outcomes, as well as to expand the nature of the outcomes categories presented above. The key point is, however, that outcomes can be many and varied--and produced by the same service or resource (e.g., public access Internet Internet

Publicly accessible computer network connecting many smaller networks from around the world. It grew out of a U.S. Defense Department program called ARPANET (Advanced Research Projects Agency Network), established in 1969 with connections between computers at the
 workstations, online databases).

As identified in Table 3, there are two additional aspects of outcomes that researchers and practitioners need to consider: the level of outcome applicability and the time dimension of the outcome itself (discussed below). It is important to note that outcome types may apply at different levels (illustrated by the columns in Table 3). For example, a learning outcome may apply to the:

* Individual user of the library's services/resources by gaining the ability to read;

* Library by having a now-literate customer consume other library services/resources and derive additional impacts/benefits;

* Community by now having a literate member who can seek and expand his/her employability; and

* State and nation through a more economically solvent solvent, constituent of a solution that acts as a dissolving agent. In solutions of solids or gases in a liquid, the liquid is the solvent. In all other solutions (i.e.  individual who contributes more to the economy through higher wages.

As such, outcomes can have multiple impacts for the individual user, library, library institution (e.g., university), system (i.e., state university system, public library system as a whole), state, and nation.

According to according to
1. As stated or indicated by; on the authority of: according to historians.

2. In keeping with: according to instructions.

 some authors, outcomes assessment implies that outcomes are predefined through a planning process in which libraries engage (Hernon & Dugan, 2002; Fraser & McClure, 2002). In this view, outcomes are established a priori through service and resource goals and objectives developed through a planning process; there is some type of pretest pre·test  
a. A preliminary test administered to determine a student's baseline knowledge or preparedness for an educational experience or course of study.

b. A test taken for practice.

 in which libraries ascertain the extent to which their services/resources meet these goals and objectives according to users; and then, after user exposure to the new and/or modified library services/resources, the library measures the impact (outcome) of the services/resources through a posttest post·test  
A test given after a lesson or a period of instruction to determine what the students have learned.
. Again, this is an oversimplification o·ver·sim·pli·fy  
v. o·ver·sim·pli·fied, o·ver·sim·pli·fy·ing, o·ver·sim·pli·fies
To simplify to the point of causing misrepresentation, misconception, or error.

 of the outcomes environment. Indeed, it is possible to have outcomes that are on different time dimensions as the rows in Table 3 demonstrate. Outcomes can be:

* Anticipated. These are the outcomes for which the library plans and by which the library intends to measure its success/failure in goals and objectives attainment. The library expects to achieve certain outcomes through its services/resources and then seeks to ascertain the extent to which its services/resources achieved the anticipated outcomes. It is important to note that anticipated outcomes can be generated by the library or imposed externally by funding agencies (e.g., the Institute of Museum and Library Services The Institute of Museum and Library Services is an independent agency of the United States federal government. It is the main source of federal support for libraries and museums within the United States. ) or governing gov·ern  
v. gov·erned, gov·ern·ing, gov·erns
1. To make and administer the public policy and affairs of; exercise sovereign authority in.

 authorities (e.g., university boards and provosts, accreditation bodies, city council, etc.). Most outcomes work to date focuses on this type of outcome assessment model.

* Emergent emergent /emer·gent/ (e-mer´jent)
1. coming out from a cavity or other part.

2. pertaining to an emergency.


1. coming out from a cavity or other part.

2. coming on suddenly.
. These are outcomes that emerge through the service/resource planning and implementation process. Such outcomes are not the immediate focus of the service/resource goals and objectives--either library or externally imposed. However, as the library develops its service/ resource plans, these additional outcomes become apparent and are incorporated into the assessment process.

* Unanticipated. Once a service/resource is in operation, there are those outcomes that derive from actual service/resource use or interaction and can be ones that neither the library nor others predicted--nor planned to assess. For example, in the early years of public library Internet connectivity, many public librarians This is a list of people who have practised as a librarian and are well-known, either for their contributions to the library profession or primarily in some other field.  anticipated that the primary outcome of Internet connectivity in their libraries would be for information seeking Information seeking is the process or activity of attempting to obtain information in both human and technological contexts. Information seeking is related to, but yet different from, information retrieval (IR). , and quickly discovered that the primary outcome was communications--the ability of users to stay in touch with family and countries (McClure & Bertot, 1998). This outcome is still prevalent prevalent

widespread occurrence.
 in public libraries, as demonstrated recently by Lance et al. (2002). By the same token In programming, a string of characters. For example, in the C expression #define MAXAMOUNT 50000, MAXAMOUNT is the token. See also token passing and authentication token.

, universities enhanced their Internet connectivity in their goal to become "wired campuses." Of the many anticipated outcomes
   associated with robust connectivity, one unintended outcome was the
   substantial use of university resources by students to access online
   music services (some later deemed illegal). A number of universities
   now block access to such services from their campuses to preserve
   their connectivity resources, as well as to limit the universities
   from liability due to various copyright infringements.

It is clear that outcomes are not always anticipated either by librarians or others with expectations for library service/resource use. By focusing on the anticipated outcomes, librarians (and others) ignore significant emergent and unanticipated outcomes--both positive and negative. Any outcomes assessment strategy, therefore, needs to incorporate the ability to discover and articulate outcomes that were not the intended focus of the services/resources--something not likely in a pre- pre- word element [L.], before (in time or space).

1. Earlier; before; prior to: prenatal.

 or posttest quasi-experimental methodology.

Outcomes Assessment Methodology Issues

Embedded Inserted into. See embedded system.  within the discussion of the types of outcomes identified above is the notion of gap identification. In an ideal situation, a library anticipates a set of outcomes from its services/resources, and the library's user community confirms those anticipated outcomes. It is more likely the case, however, that outcomes reside within a zone of tolerance in which library intended outcomes and user assessment of those outcomes are within an acceptable range--or minimal gap. There may be multiple gaps or zones of tolerance--the library, the university, the user--all of these constituencies comprise the outcomes space within which the library's services/resources reside. There is a need to explore further the notion of gap analysis in outcomes assessment activities such as is being done with service quality research activities (Cook et al., 2001).

Another critical factor in outcomes assessment is the ability to isolate isolate /iso·late/ (i´sah-lat)
1. to separate from others.

2. a group of individuals prevented by geographic, genetic, ecologic, social, or artificial barriers from interbreeding with others of their kind.
 the actual impact of library services/resources on the user community. As an example, we use one of Smith's (2000) institutional learning outcomes for university libraries, Develop Attitudes of Openness, Flexibility, Curiosity Curiosity

so assured of wife’s fidelity, asks friend to try to corrupt her; friend is successful. [Span. Lit.: Don Quixote]

Cupid and Psyche

her inquisitiveness almost drives him away forever. [Gk. Myth.
, Creativity, and an Appreciation of the Value of a Broad Perspective. Let us create a formula based on general linear model (GLM GLM Global Language Monitor
GLM Global Marine (stock symbol)
GLM Graduated Length Method (ski instruction)
GLM Good Looking Mom (used in pediatric practices)
GLM God Loves Me
) analysis techniques for which one might use analysis of variance The discrepancy between what a party to a lawsuit alleges will be proved in pleadings and what the party actually proves at trial.

In Zoning law, an official permit to use property in a manner that departs from the way in which other property in the same locality
 (ANOVA anova

see analysis of variance.

ANOVA Analysis of variance, see there
) or multiple regression Multiple regression

The estimated relationship between a dependent variable and more than one explanatory variable.
 to analyze an·a·lyze
1. To examine methodically by separating into parts and studying their interrelations.

2. To separate a chemical substance into its constituent elements to determine their nature or proportions.


Dependent Variable = Students develop attitudes of openness, flexibility, curiosity, creativity, and an appreciation of the value of a broad perspective

Independent Variable (s) = Particular library services/resources + courses + student extracurricular activities + ... +?

In quantitative terms, researchers as well as librarians and university officials will want to know what library services/resources contribute, in what way, and how much, to the student body's development of attitudes of openness, flexibility, curiosity, creativity, and an appreciation of the value of a broad perspective. In other words, the assessors of program outcomes will want to know which library services/resources contribute to the attainment of the outcome, as well as by how much, and in what way(s).

Needless to say, it is near impossible to quantify Quantify - A performance analysis tool from Pure Software. , much less isolate, a university library's contribution toward the outcome identified above. Simply put, student "attitudes" are formed by any number of campus activities in which students might engage--from coursework coursework

work done by a student and assessed as part of an educational course

Noun 1. coursework - work assigned to and done by a student during a course of study; usually it is evaluated as part of the student's
 to mentoring by faculty to clubs to which the student may belong, etc. Library services/resources do not exist in a vacuum and measurement techniques will need to take these external and likely contributing factors into account. This is particularly the case when outcomes are presented as an institutional (i.e., university) objective but measurement occurs locally (i.e., at the library). To ignore the other university activities that likely contribute to the outcome would yield an incomplete picture of library contributions to outcome attainment at best and distort the library's contribution to outcomes attainment at worst.


Libraries now reside in a complex service environment--one that requires that they provide traditional services and resources such as a physical space, print material, and face-to-face (jargon, chat) face-to-face - (F2F, IRL) Used to describe personal interaction in real life as opposed to via some digital or electronic communications medium.  reference, as well as network-based services such as Web-based collections, online databases, and virtual reference (Bertot, 2000; McClure, Fraser et al., 2002). Libraries are, therefore, providing more services and resources through multiple delivery methods. Moreover, while these traditional and network-based services are perhaps related and similar in function, they differ vastly in a number of significant ways such as the infrastructure required to deliver the services, the ways in which users or user communities access the services, the skills and knowledge required by uses or user communities to use the services, the reach and range of the services, the ways in which libraries manage such services, and the skills required within the library to deliver and access such services.

The networked environment is an intricate series of networks that interconnect (1) To attach one device to another.

(2) A physical port (plug, socket) or wireless port (transmitter, receiver) used to attach one device to another.
 in such a way that users never completely know the route they took to retrieve To call up data that has been stored in a computer system. When a user queries a database, the data are retrieved into the computer first and then transmitted to the screen.

(language) Retrieve
 the information or resource that they sought. As Smith and Rowland Row·land   , F(rank) Sherwood Born 1927.

American chemist who shared a 1995 Nobel Prize for his work on the chemical processes involved in the formation and decomposition of ozone.
 (1997, p. 170) note, local users can access local resources, remote users can access local resources, and local users can access remote resources. Such a multifaceted mul·ti·fac·et·ed  
Having many facets or aspects. See Synonyms at versatile.

Adj. 1. multifaceted - having many aspects; "a many-sided subject"; "a multifaceted undertaking"; "multifarious interests"; "the multifarious
 environment--one in which the library is the resource or the library is the resource gateway for local or remote users--creates a complex environment for measurement. Data collection and research activities, therefore, need to reflect this complexity and multidimensionality of electronic networks. They also need to recognize the need for collecting a range of data for use at the organizational (library), institutional (university, county government), state, or national levels.

The implications for outcomes assessment due to this environment are numerous. They include, but are not limited to:

* Many library network-based services are not truly the services of that library. For examples, libraries license the use of such vendor-based services as EbscoHost and ScienceDirect. As such, they do not own, but rather lease, such "collections." Therefore, any measurement activities that assess the outcomes of library services need to take into account that libraries do not have direct control over such services. Thus, how best to "count" basic networked service interactions is complex (Shim & McClure, 2002).

* Users can and do access network-based services from numerous locations using a wide range of technology and telecommunications services In telecommunication, the term telecommunications service has the following meanings:

1. Any service provided by a telecommunication provider.

. Each of these variations in access can and does affect the experience of the user--and these variations are beyond the purview The part of a statute or a law that delineates its purpose and scope.

Purview refers to the enacting part of a statute. It generally begins with the words be it enacted and continues as far as the repealing clause.
 of the library. Indeed, some library users do not access the library physically.

* The type of interaction that users can have with a library's network-based services and resources versus a library's physical services and resources can be quite different. Virtual users do not "browse (1) To view the contents of a file or a group of files. Browser programs generally let you view data by scrolling through the documents or databases. In a database program, the browse mode often lets you edit the data. See Web browser.  the stacks" as do physical users. Moreover, not all of a library's collection is available in digital format; thus, virtual users are often limited to certain types of resources and services. In some cases, a service or resource is only available in electronic format. Another service that libraries provide increasingly is user training (e.g., use of technology, use of databases, other). Online tutorials differ greatly from in-person adj. 1. undertaken by an individual in person; as, an in-person appearance s>.

Adj. 1. in-person - an appearance carried out personally in someone else's physical presence; "he carried out the negotiations in person"; "a
 training in a library lab setting.

These issues suggest that research activities cannot regard all users as the same, nor can they treat all library services and/or service delivery modes as the same.


As identified earlier in this article, the discussion of outcomes and outcomes measurement in libraries and related information organizations is in its infancy infancy, stage of human development lasting from birth to approximately two years of age. The hallmarks of infancy are physical growth, motor development, vocal development, and cognitive and social development. . There are more questions about outcomes and outcomes assessment than there are adequate answers at this point. Indeed, it may be that at some point in the future, researchers will determine that outcomes and outcomes assessment have little utility in library organizational and services evaluation because of problems of implementations--especially in a networked environment. The research questions posed in the beginning of this article only scratch the surface of those that require attention. As the research and library communities move forward in studies related to outcomes and outcomes assessment, there is a need to make explicit some assumptions, considerations, and concerns.

Need for Empirical Research

At present, there is no lack of opinions and views related to outcomes and outcomes assessment. But, there is a lack of empirical research related to outcomes, outcomes assessment, and conceptual frameworks For the concept in aesthetics and art criticism, see .

A conceptual framework is used in research to outline possible courses of action or to present a preferred approach to a system analysis project.
 that would help practitioners to understand better the basis for, application of, and use of outcomes assessment. One empirical effort was the Association of Research Libraries' sponsored E-metrics study (Association of Research Libraries, 2002). One aspect of the E-Metrics project explored the role of the academic library in its contribution to achieving institutional outcomes. Findings from the E-Metrics study suggest (Fraser and McClure, 2002):

* The inputs-outputs model for library assessment (which produced the statistics and measures in the report Measures and Statistics for Research Library Networked Services: Procedures'and Issues) may not be linked easily to demonstrating the library's role in accomplishing institutional outcomes.

* Many of the library's activities, resources, and services are combined with other institutional activities, resources, and services in such a way that parsing See parse.

parsing - parser
 out only the library's contribution to institutional outcomes is extremely difficult.

* The process by which libraries are involved in the identification of and agreement to these institutional outcomes is not clear.

* There is widespread confusion as to what an "institutional outcome" is and how such outcomes "fit" into traditional assessment procedures.

* Increasingly, academic accreditation agencies are considering the use and appropriateness of "institutional outcomes" as a means to assess the degree to which the organization determines what end products should result from organizational activities.

* The issue of nonexistent non·ex·is·tence  
1. The condition of not existing.

2. Something that does not exist.

 or inconsistent Reciprocally contradictory or repugnant.

Things are said to be inconsistent when they are contrary to each other to the extent that one implies the negation of the other.
 incomparable (mathematics) incomparable - Two elements a, b of a set are incomparable under some relation <= if neither a <= b, nor b <= a.  usage statistics provided by external information content providers (vendors) is a major stumbling block stum·bling block
An obstacle or impediment.

stumbling block

any obstacle that prevents something from taking place or progressing

Noun 1.
 for libraries to gauge rapidly increasing use of electronic resources by research library users, thus making it difficult to use the data as sources for establishing library outcomes.

To a large degree, the E-metrics study found that academic libraries are not well prepared to demonstrate the extent to which they contribute to the organization's accomplishment of institutional outcomes. Moreover, libraries are looking at outcomes at the library level when, in fact, they reside within an institutional context. Nonetheless, there appears to be considerable interest in identifying and measuring the degree to which the academic library does contribute to accomplishing such institutional outcomes.

One of the products from the E-metrics study was a proposal for additional research in the area of outcomes, "Identifying and Measuring Library Activities/Services Related to Academic Institutional Outcomes" (McClure et al., 2002). The proposal outlined a conceptual framework for understanding outcomes assessment that has yet to be tested. In fact, there are few efforts to propose conceptual frameworks (such as that offered in Figure 2 earlier in this article) to help researchers lodge such research in the broader frame of evaluation theory. Thus, empirical research projects are necessary to understand better and describe outcomes and outcomes assessment--regardless of the type of outcomes being investigated.

Need for Multimethod Approaches

From the issues identified and discussed in this article, it is clear that research endeavors cannot treat all users as the same, all delivery of library services in the networked environment as the same, nor should they lump generically library services for assessment purposes. The differences between online and physical services/resources are real and not comparable (Bertot, McClure, & Davis, 2002; Shim et al., 2002; Bertot, 2001). Research efforts that do not separate network-based library services and resources from physical services/resources--such as work by Hernon and Dugan (2002) and Hernon and Whitman Whitman, town (1990 pop. 13,240), Plymouth co., SE Mass., S of Boston; settled c.1670, set off from Abington and inc. 1875. It is an industrial town that manufactures shoes, plastics, foundry products, and textile machinery. The Toll House (1709) is restored.  (2001)--miss substantive Substantive may refer to:

In grammar:
  • a noun substantive, now also called simply noun
  • a verb substantive, a verb like English "be" when expressing existence (in contrast to use as a copula)
In law:
 differences and, ultimately, the ability to determine library service and resource quality and outcomes.

While there is a need to focus much research on outcomes, such research should not occur at the exclusion of promising work in other areas of assessment. For example, there is a range of other efforts currently underway to assist the library community in the assessment of services and resources. These other efforts focus on:

* Service discrepancies among different stakeholder stakeholder n. a person having in his/her possession (holding) money or property in which he/she has no interest, right or title, awaiting the outcome of a dispute between two or more claimants to the money or property.  groups (e.g., LibQUAL+);

* Inputs analysis;

* Outputs analysis;

* Cost/benefit analysis;

* Performance measures (that can combine inputs and outputs); and

* Quality standards.

The issue is not which of these, or others, is the best approach for libraries. The issue is better stated as: Given the needs of the library, the resources available to the library for assessment, the assessment activity objectives, and the staff skills available--what methods for assessment will be most useful and successful?

Oftentimes of·ten·times   also oft·times
Frequently; repeatedly.

Adv. 1. oftentimes - many times at short intervals; "we often met over a cup of coffee"
frequently, oft, often, ofttimes
, a combination of methods is best as each method has its own strengths and weaknesses. For example the recently published manual Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services: Guidelines guidelines, a set of standards, criteria, or specifications to be used or followed in the performance of certain tasks.
 and Procedures (McClure et al., 2002) offers some thirty-five performance measures related to the provision of digital reference services Digital reference is a service by which library reference service is conducted online, and the reference transaction is a computer-mediated communication.

The word "reference" in this context refers to the task of providing assistance to library users in finding information,
 as well as six suggested areas for quality standards. These measures and quality standards are a first comprehensive effort for assessing digital reference services. Inclusion of cost-benefit analysis cost-benefit analysis

In governmental planning and budgeting, the attempt to measure the social benefits of a proposed project in monetary terms and compare them with its costs.
, service quality approaches, and outcomes assessment could also be incorporated with the approach described in the manual. As methods are integrated and expanded, however, the cost and level of effort required to conduct the assessment increases.

Multimethod approaches are also likely to be most useful given the range of situational factors that affect individual organizations and libraries. The type of assessment required for a small rural public library that is a member of a large regional cooperative with few staff and resources is likely to vary considerably from a large academic library that purchases an extensive collection of electronic resources, has talented technology staff, and provides customized networked services to its academic community.

In the experience of these authors, one evaluation method does not fit all types of libraries. The promotion of one particular method by some at the expense of all other methods does not reflect the complexity of situational factors as they relate to assessment in a library context. Nor might a single approach provide reliable and valid research results in all library organizational settings--there is a need for flexibility in methods and execution for libraries to engage successfully in evaluation activities.

Impact of the Networked Environment

As discussed earlier in this article, assessment of library services and resources--regardless of method--needs increasingly to consider the role and impact of electronic services and resources. Assessment of electronic services and resources, especially in a networked environment, involves a very different set of circumstances than those in the print and physical environment (McClure & Bertot, 2001). Assessment in a networked environment raises the following unique issues:

* The technology for the delivery of these services and resources is constantly changing, which affects what services/resources a library can deliver and how such delivery might occur. This also affects the methods, approaches, and software that enable the "tracking" of such services/resources;

* There is a need to have comparative (and valid) measures for traditional versus electronic-based services and resources, e.g., is a physical visit to the library (turnstile count) comparable to a visit to the library's Web site?

* Oftentimes, libraries contract (license) for networked services or do not control their own technology infrastructure (e.g., servers), thus they cannot easily obtain transactional data describing services and resource provision;

* The increased reliance on consortia and other group arrangements for the purchase and delivery of electronic services and resources blurs actual costs for these services and resources as well as how these services and resources are being used;

* Related to the above, leased services/resources are not truly the services/resources of the library. Rather, the library acts as a gateway to such services/resources. Outcomes assessment activities need to consider the most appropriate strategies for evaluating the outcomes of services/resources that are technically not the library's; and

* Librarian (1) A person who works in the data library and keeps track of the tapes and disks that are stored and logged out for use. Also known as a "file librarian" or "media librarian." See data library.

(2) See CA-Librarian.
 skills necessary to conduct outcomes and other types of assessment in the networked environment are significant and require constant enhancement given the changing nature of that environment.

Ease of Use

One major concern regarding the development of outcomes assessment methods is the ability of libraries to implement the recommended approaches, data collection activities, interpret the results, and use the findings to inform library resource/service planning efforts as well as external library stakeholders Stakeholders

All parties that have an interest, financial or otherwise, in a firm-stockholders, creditors, bondholders, employees, customers, management, the community, and the government.
. Thus, it is imperative that researchers engaged in the development of outcomes assessment tools consider the degree to which the methods are practical and feasible for day-to-day day-to-day
1. Occurring on a routine or daily basis: the day-to-day movements of the stock market.

 use in libraries and other information organizations. As this article indicates, however, the ability of researchers to meet this ease-of-use burden remains unclear as outcomes assessment requires complex indicators and methods.

To some degree, outcomes assessment is a theory in search of a practice. From the earliest days of the development of various output measurement manuals in the early 1980s, many practitioners remain unconvinced that the amount of time and resources necessary for conducting assessments--of any type--is worth the bother. Despite a major effort at making Output Measures for Public Libraries (Van House et al., 1987) easy to use, practical, and useful for library decision-making decision-making,
n the process of coming to a conclusion or making a judgment.

decision-making, evidence-based,
n a type of informal decision-making that combines clinical expertise, patient concerns, and evidence gathered from
, some practitioners found that this, and similar assessment manuals, required too much time and effort to implement.

A major difficulty with outcomes assessment is that understanding outcomes, developing approaches to use outcomes as an assessment technique, training staff to be able to implement outcomes assessment, and then using the results for decision-making can be time-consuming time-con·sum·ing
Taking up much time.


taking up a great deal of time

Adj. 1.
 and difficult. Despite attempts for practical outcomes assessment manuals (e.g., Florida Florida, state, United States
Florida (flôr`ĭdə, flŏr`–), state in the extreme SE United States. A long, low peninsula between the Atlantic Ocean (E) and the Gulf of Mexico (W), Florida is bordered by Georgia and
 State Library, 2000) much work remains to be done to make outcomes assessment a practical, valid, and easy approach that will be embraced by the library community. Those developing such manuals need to work toward methodologies that are feasible, cost effective, and actionable Giving sufficient legal grounds for a lawsuit; giving rise to a Cause of Action.

An act, event, or occurrence is said to be actionable when there are legal grounds for basing a lawsuit on it.
 with relative ease. It is unclear at this time how well outcomes assessment--as a method--can be demystified and integrated easily into an ongoing library evaluation program.

Local Judgment Calls

Missing from a number of discussions about the importance of outcomes and outcomes assessment is the discussion of what are the "correct," "appropriate," or "right" outcomes for a particular library setting. Thus, the research question here is: What factors within the library and the local community affect the judgment of when a service is "good enough"? The issue becomes more complicated if the library has to accommodate its outcomes with those of a larger institution (university or city government). Moreover, once there is a selection of a particular outcome as appropriate for that setting, "how much" of that outcome does the library need to obtain to claim "success"? Similar issues occur in other types of assessment methods. The concern, however, is in method replacing value judgments on the part of the users of these methods.

As an example, the performance measure "correct answer fill rate," offers no guidance unless we know if a 50 percent correct answer fill rate is acceptable for a particular library or if a 75 percent correct answer fill rate is desired instead. Having a performance measure, or having an outcome, is not the same as agreeing on the quality or performance level that a library wishes to achieve on that outcome or performance level. It is the responsibility of the individual library or organization to develop a process that results in agreement that the quality standard for correct answer fill rate might be something like, "reference staff will answer quick fact and bibliographic bib·li·og·ra·phy  
n. pl. bib·li·og·ra·phies
1. A list of the works of a specific author or publisher.

 questions correctly 70 percent of the time."

The quality level or "standard" that a library or organization accepts as a benchmark A performance test of hardware and/or software. There are various programs that very accurately test the raw power of a single machine, the interaction in a single client/server system (one server/multiple clients) and the transactions per second in a transaction processing system.  by which to judge itself depends on:

* Institutional and library goals and objectives (either developed internally or imposed externally);

* Stated priorities among those objectives (libraries cannot be all things to all people all the time);

* Resource allocations resource allocation Managed care The constellation of activities and decisions which form the basis for prioritizing health care needs  among goals, objectives, and services;

* Staff knowledge and skills in providing information services; and

* Other situational factors at play at an individual setting.

The easy part of evaluation may be identifying and validating val·i·date  
tr.v. val·i·dat·ed, val·i·dat·ing, val·i·dates
1. To declare or make legally valid.

2. To mark with an indication of official sanction.

 the performance measures, outcome, or various statistics as indicators of information services. The hard part may be the development of a process within the organization to agree on the quality standards for those outcomes that are acceptable, or are the target, for that particular organization. In short, outcomes assessment has little utility without accompanying quality standards, and these require judgment calls on the part of the users of such methods.

Recognizing the Political Context of Outcomes Assessment

Evaluation, overall, has significant political overtones beginning with the determination of what to evaluate, how it will be evaluated, what outcomes are most important, what measures to use, how to interpret findings, and how best to report or present the findings. Noticeably no·tice·a·ble  
1. Evident; observable: noticeable changes in temperature; a noticeable lack of friendliness.

2. Worthy of notice; significant.
 absent from a number of the discussions of outcomes assessment are considerations related to the political environment in which the library exists. A determination of "appropriate" outcomes by the larger institution can eliminate effectively the ability of the library to demonstrate how it supports those outcomes. In addition, these political considerations are compounded because of the network and technology infrastructure through which library services and resources are provided. For example, a public library that is dependent on a city's technology infrastructure for its network and resultant This article is about the resultant of polynomials. For the result of adding two or more vectors, see Parallelogram rule. For the technique in organ building, see Resultant (organ).

In mathematics, the resultant of two monic polynomials
 services has a more complicated political context for determining and measuring outcomes than a library that controls its own technology.

Dealing with the various political issues related to outcomes and outcomes assessment in a networked environment is new to most libraries. In addition, outcomes assessment in the highly charged technological infrastructure of many organizations requires the evaluator to have a range of skills and knowledge not needed in the nonnetworked environment. Research is needed to shed light on those serious issues.


As suggested in the key research questions posed earlier in this article, it is necessary to view research related to outcomes and outcomes assessment in the larger context of overall library and information services assessment. Outcomes assessment is but one method that has the potential to help library managers make better decisions regarding the provision of information services and resources. At present, however, there are many issues and problems with practical implementations of the approach. While there certainly is potential for developing outcomes assessment, much work--especially empirical research--is necessary.

As discussed previously, use of the terms outcomes, performance measures, quality standards, service quality, etc. continue to be disputed. As Kyrillidou (2002) concludes in a recent article, not only is there a need for more dynamic models to describe libraries and users, "we lack an adequate shared understanding of how outcomes are defined" (p. 45). The framework outlined in this article (see Figure 2) is one effort to help clarify the use of these terms and provide a means to understand their relationship to each other. Without better agreement among researchers about the use of these terms there is likely to be limited implementation of these various assessment techniques. Indeed, at a recent workshop conducted by the authors, a participant commented, after a review of the various assessment methods and approaches, that researchers and writers studying outcomes were like kids playing in a sandbox--and that clear, practical, usable USable is a special idea contest to transfer US American ideas into practice in Germany. USable is initiated by the German Körber-Stiftung (foundation Körber). It is doted with 150,000 Euro and awarded every two years.  evaluation methods and approaches are not available to practitioners.

Also lacking from many library evaluation approaches is adequate recognition of how the networked environment affects services and resource provision. As shown earlier in Figure 1, the way online services are accessed raises a range of evaluation issues that require substantial attention. A key theme throughout this article is that outcomes assessments, as well as other types of library assessment, have yet to address factors and issues arising from the networked environment that seriously complicate com·pli·cate  
tr. & intr.v. com·pli·cat·ed, com·pli·cat·ing, com·pli·cates
1. To make or become complex or perplexing.

2. To twist or become twisted together.

 valid assessment approaches. Indeed, little of the outcomes assessment work to date considers the evaluation of network-based services or resources--and how those services/resources differ from traditional library services or resources. And yet, libraries are increasing the network-based services and resources that they provide.

There are a number of possible next steps for researchers working in the area of outcomes assessment:

* Develop a better understanding of outcomes and how outcomes "fit" into the range of evaluation models. This could be done by comparing the strengths and weaknesses of the models as shown in Table 2 and by reviewing the considerations given in this article that the models do not address currently.

* Specify and rethink re·think  
tr. & intr.v. re·thought , re·think·ing, re·thinks
To reconsider (something) or to involve oneself in reconsideration.

 college and university accreditation documents to better determine what they mean by outcomes assessment and the process by which such assessment should be done. The review of such accreditation documents by Gratch-Lindauer (2002) clearly identifies confusion and lack of understanding by these accreditation agencies concerning outcomes assessment.

* Involve library practitioners in research related to outcomes to obtain their knowledge, views, and experiences in evaluation.

* Conduct empirical research on the validity of Table 3 in this article and determine the degree to which outcomes are similar or different in a traditional (print and face-to-face service provision) environment versus a networked environment.

* Conduct empirical research that determines the extent to which outcomes are generic and, thus, obtainable and comparable across a range of libraries and library types based on library situational and other factors.

* Sponsor a small meeting or symposium symposium

In ancient Greece, an aristocratic banquet at which men met to discuss philosophical and political issues and recite poetry. It began as a warrior feast. Rooms were designed specifically for the proceedings.
 in which key researchers and practitioners could debate and discuss a range of issues and concerns regarding outcomes assessment. Such a meeting could help focus attention on how best to attack outcomes assessment as an evaluation method.

These are only a few efforts possible to begin to address some of the research questions identified in this article. Until such steps are taken and a range of research activities initiated, outcomes assessment will continue to be an idea in search of both a theory and a practice.

There is much work yet to do in translating outcomes assessment and other evaluation approaches into practical, useful, and valid assessment techniques in the networked environment. The track record for the degree to which practitioners have embraced a program of ongoing evaluation (regardless of the method) is extremely low. Lakos (1999) identified the need for a "culture of assessment" within libraries years ago--and it is still lacking in large part. Needed now is more empirical research related to these evaluation approaches; evaluation methods that recognize and understand the role of the networked environment in the provision of services and resources; partnerships with libraries and related organizations to test, refine, and validate To prove something to be sound or logical. Also to certify conformance to a standard. Contrast with "verify," which means to prove something to be correct.

For example, data entry validity checking determines whether the data make sense (numbers fall within a range, numeric data
 practical and doable approaches; and a commitment to developing multiple evaluation methods that can work successfully together.
Table 1. Selected Definitions of Outcomes and Outcomes Assessment. *

Citation                                Definition of Outcome

Zweizig, D., Johnson, D., Robbins, J.,  "Outcomes--how things
& Besant, J. (1994). TELL IT!           changed for the
Evaluation sourcebook and training      community" (p. 104).
manual. Madison, WI: University of
Wisconsin, Madison, School of
Library and Information

United Way. (1996). Measuring program   "Outcomes are benefits for
outcomes: A practical approach.         participants during or after
Alexandria, VA: United Way.             their involvement with a
                                        program. Outcomes may relate to
                                        knowledge, skills, attitudes,
                                        values, behavior, condition, or
                                        status" (p. xv).

Himmel, E., & Wilson, W. J. (1998).     "... a service response is what
Planning for results: A public          a library does for, or offers
library transformation process: The     to, the public in an effort to
how-to manual. Chicago: American        meet specific community needs.
Library Association.                    Service responses represent the
                                        gathering and deployment of
                                        critical resources to produce a
                                        specific public benefit or
                                        result" (p. 51-52).

Himmel, E., & Wilson, W.J. Planning     "Evaluation is the process
for results: A public library           used to measure the
transformation process: The             performance of a
guidebook, Chicago: American            service against some
Library Association.                    pre-determined criterion
                                        to see how well or poorly
                                        the service has been
                                        performed" (p. 35).

Institute of Library and Museum         Outcomes are "benefits or
Services. (2000). Perspectives on       changes for individuals
outcome based evaluation for            or populations during or
libraries and museums.                  after participating in
Washington, D.C.: Institute of          program activities ..."
Library and Museum Services.            that are measured against
                                        predetermined criteria
                                        (p. 20).

Free Library of Philadelphia.           Outcomes are "the quality
(2000, November 30). Free Library       of things produced" as
of Philadelphia: Performance            related to the goals set
management final report.                forth early in the project's
Philadelphia, PA: Price                 development (E, p. 1).
Waterhouse Coopers.

Association of College and              Focus "on the achievement of
Research Libraries. (2000).             outcomes that have been
Standards for college libraries         identified as desirable in
2000 edition. Chicago: American         the library's goals and
Library Association, ACRL               objectives. It identifies
College Libraries Section               performance measures,
Standards Committee.                    such as proficiencies,
Available at: http://www.               that indicate how well the        library is doing what it
                                        has stated it wishes to do."

Bertot, J. C., McClure, C. R., &        "An outcome measure is
Ryan, J. (2001). Statistics and         explicitly tied to the
performance measures for public         libraries goals, objectives,
library networked services.             and planning process. A
Chicago: American Library               good outcome measure provides
Association.                            data that tells a library
                                        manager if a specific
                                        library objective
                                        has been achieved" (p. 66).

Hernon, P., & Dugan, R. E. (2002).      "Outcomes assessment deals
An action plan for outcomes assessment  with academic institutions
in your library. Chicago:               providing evidence that they
American Library Association.           are meeting their educational
                                        mission. In the case of the
                                        library, outcomes focus on how
                                        library users changed as a
                                        result of their contact with
                                        the library and its resources,
                                        services, and programs" (p. 2).

Fraser, B. T., & McClure, C. R.         An outcome is a "clearly
(2002, forthcoming). Toward a           identified result or end
framework for assessing                 product that occurs as a
library and institutional               consequence of individual
outcomes. Portal: Libraries             or combined activities from
and the Academy, 2(4), 505-528.         units of the institution.
                                        It is a preferred or desired
                                        state and ideally clarifies
                                        specific expectations of
                                        what should be products
                                        from the institution."

* The authors wish to thank Lara Rudolph for her assistance in
compiling the citations in this table.

Table 2. Selected Outcome Models. *

Citation                                       Model

Kyrillidou, M. (2002). From               The Linear Model
input and output measures to
quality and outcome measure,
or, from the user in the life of          The Cyclic Model
the library to the library in the
life of the user. The Journal of
Academic Librarianship, 28, pp.           The Spiral Swirl

Bertot, J. C., & McClure, C. R.            Outcomes and
(2003). Outcomes assessment in            Performance
the networked environment:                Measurement in
Research questions, issues,               Libraries
considerations, and moving
forward. Library Trends 51:4.

Hernon, P. (2002). Outcomes               Aggregate Factors
are key but not the whole                 Approach
story. The Journal of Academic
Librarianship, 28, pp. 54-55.

King et al. (2003). Library               Derived Measures
economic measures. Library
Trends, 51 (3).

Fraser, B. T., & McClure,                 Preliminary
C. R. (2002). Toward a                     Framework of the
framework for assessing                   Outcomes Assessment
library and institutional                 Process
outcomes. Portal: Libraries
and the Academy, 2,
pp. 505-528.

Citation                                       Summary

Kyrillidou, M. (2002). From               Based on the "assumption
input and output measures to              that inputs have a
quality and outcome measure,              direct relation to
or, from the user in the life of          outputs, which, in turn,
the library to the library in the         relates to quality
life of the user. The Journal of          and outcomes" (p. 44).
Academic Librarianship, 28, pp.
42-46.                                    Acknowledges that user
                                          transactions have
                                          "multiple dimensions
                                          of inputs, output, and
                                          quality and outcome
                                          elements from multiple
                                          interactive and reflective
                                          perspectives" (p. 44-45).

                                          "[T]ries to introduce
                                          the notion of motion
                                          depicting a more
                                          dynamic and flexible
                                          model, moving users
                                          and information
                                          resources into a
                                          spiral swirl up and down
                                          into the depths of
                                          knowledge, exploration,
                                          and experience" (p. 45).

Bertot, J. C., & McClure, C.R.            "Inputs and outputs form
(2003). Outcomes assessment in            the basis for service
the networked environment:                quality and outcomes
Research questions, issues,               assessment activities.
considerations, and moving                Outcomes assessment seeks
forward. Library Trends 51:4.             to determine the impact of
                                          the library's
                                          (again, outputs) on the
                                          library service and
                                          resource users. One
                                          cannot have outcomes
                                          measures without
                                          measures of outputs--
                                          libraries need to know
                                          what investments (inputs)
                                          produce what services
                                          (outputs) in order to
                                          determine the perceived
                                          quality (quality assessment)
                                          and impacts (outcomes) of
                                          those services/resources"
                                          (p. X).

Hernon, P. (2002). Outcomes               Sees "library planning and
are key but not the whole                 decision making as
story. The Journal of Academic            revolving around service
Librarianship, 28, pp. 54-55.             quality and its companion
                                          concept, satisfaction, and
                                          outcomes (or more precisely
                                          outcomes assessment). Inputs
                                          and outputs support the
                                          accomplishment of service,
                                          quality, satisfaction, and
                                          an assessment plan" (p. 54).

King et al. (2003). Library               "Measures are designed to
economic measures. Library                serve the perspectives of
Trends, 51 (3).                           library staff management,
                                          library users, the
                                          fundors of the library and
                                          the higher order community
                                          served by the library" (X).
                                          "Outcomes are best determined
                                          by relating them to the
                                          purposes for which
                                          the information is
                                          obtained ..." (X).

Fraser, B. T., & McClure,                 Model depicts "a basic
C.R. (2002). Toward a                     process by which an
framework for assessing                   academic research library
library and institutional                 helps meet the goals of
outcomes. Portal: Libraries               particular departments and
and the Academy, 2,                       functional units within a
pp. 505-528.                              university, which in turn
                                          contribute to institutional
                                          goals, while acknowledging
                                          that libraries may also
                                          contribute more
                                          directly" (p. 519).

* The authors wish to thank Kim A. Thompson for her assistance
in compiling the citations in this table.

Table 3. Outcomes and Outcome Levels.

Outcome                               Outcome Level
Time Dimension  User  Library  Institutional   System  State  National



Association of Research Libraries. (2002). Measures for electronic resources (e-metrics). Unpublished announcement. Washington Washington, town, England
Washington, town (1991 pop. 48,856), Sunderland metropolitan district, NE England. Washington was designated one of the new towns in 1964 to alleviate overpopulation in the Tyneside-Wearside area.
, D.C. Retrieved December December: see month.  12, 2002, from http://

Bertot, J. C. (2000). Libraries on the information highway: Issues and lessons learned. In P.D. Fletcher Fletcher may refer to one of the following: Ideas and companies
  • A fletcher makes arrows, see fletching.
  • Fletcher School of Law and Diplomacy, the graduate school of international relations of Tufts University, located in Medford, Massachusetts.
 & J. C. Bertot (Eds.), World libraries on the information superhighway (1) A generic name for the Internet.

(2) A proposed high-speed communications system that was touted by the Clinton/Gore administration to enhance education in America in the 21st century. Its purpose was to help all citizens regardless of their income level.
 (pp. 288-300). Hershey Hershey, uninc. city (1990 pop. 11,860), Dauphin co., S central Pa.; founded 1903 as a planned community built for workers at the Hershey Corp., the chocolate manufacturer that remains its largest employer. , PA: Idea Group Publishing.

Bertot, J. C. (2001). Measuring service quality in the networked environment: Approaches and considerations. Library Trends, 49(4), 758-775.

Bertot, J. C., McClure, C. R., & Davis, D. M. (2002) Developing a national data collection model for public library networked statistics and performance measures: Interim report. Washington, D.C.: Institute of Museum and Library Services.

Bertot, J. C., McClure, C. R., & Ryan, J. (2001). Statistics and performance measures for public library networked services. Chicago Chicago, city, United States
Chicago (shĭkä`gō, shĭkô`gō), city (1990 pop. 2,783,726), seat of Cook co., NE Ill., on Lake Michigan; inc. 1837.
: American Library Association American Library Association, founded 1876, organization whose purpose is to increase the usefulness of books through the improvement and extension of library services. .

Cook, C., & Heath, F. M. (2001). Users' perceptions of library service quality: A LibQUAL+ qualitative qualitative /qual·i·ta·tive/ (kwahl´i-ta?tiv) pertaining to quality. Cf. quantitative.


pertaining to observations of a categorical nature, e.g. breed, sex.
 study. Library Trends, 49(4), 538-584.

Cook, C., Heath, F. M., Thompson Thompson, city, Canada
Thompson, city (1991 pop. 14,977), central Man., Canada, on the Burntwood River. A mining town, it developed after large nickel deposits were discovered in the area in 1956.
, B., & Thompson, R. L. (2001). The search for new measures: The ARL LibQUAL+ study--A preliminary report. Portal: Libraries and the Academy, 1(1), 103-112.

Dresang, E. T., & Gross, M. (2001). Evaluating children's resources and services in a networked environment. In C. R. McClure & J. C. Bertot, (Eds.), Evaluating networked information services: Techniques, policy, and issues (pp. 23-44). Medford Medford.

1 City (1990 pop. 57,407), Middlesex co., E Mass., a residential and industrial suburb of Boston, on the Mystic River; settled 1630, inc. as a city 1892. Wax, paper, clothing, and furniture are among its products.
, NJ: Information Today.

Florida State Library. (2000). Workbook work·book  
1. A booklet containing problems and exercises that a student may work directly on the pages.

2. A manual containing operating instructions, as for an appliance or machine.

: Outcome measurement of library programs. Tallahassee Tallahassee (tăləhăs`ē), city (1990 pop. 124,773), state capital and seat of Leon co., NW Fla.; inc. 1825. Tallahassee is a wholesale trade and distribution center for the surrounding lumber, livestock, and agricultural area. : Florida Department Florida is a department (departamento) of Uruguay. Population and Demographics
As of the census of 2004, there were 68,181 people and 21,938 households in the department. The average household size was 3.1. For every 100 females, there were 100.4 males.
 of State, Division of Library and Information Services.

Fraser, B., & McClure, C. R. (2002). Toward a framework for assessing library and institutional outcomes. Portal: Libraries and the Academy, 2(4), 505-528.

Gratch-Lindauer, B. (2002). Comparing the regional accrediation [sic Latin, In such manner; so; thus.

A misspelled or incorrect word in a quotation followed by "[sic]" indicates that the error appeared in the original source.
] standards: Outcomes assessment and other trends. Journal of Academic Librarianship, 28(1-2), 14-25.

Hernon, P. (2002). Outcomes are key but not the whole story. Journal of Academic Librarianship, 28(1-2), 54-55.

Hernon, P., & Dugan, R. E. (2002). An action plan for outcomes assessment in your library. Chicago: American Library Association.

Hernon, P., & Whitman, J. R. (2001). Delivering satisfaction and service quality: A customer-based approach for libraries. Chicago: American Library Association.

Institute of Museum and Library Services (IMLS). (2000). Perspectives on outcome based evaluation for libraries and museums. Washington, D.C.: Institute of Museum and Library Services. Retrieved December 4, 2002, from <>.

King, D. W., Boyce Boyce may refer to:
  • Christopher John Boyce, American who sold spy satellite secrets to the USSR.
  • James Petigru Boyce, theologian and Southern Seminary founder.
  • Max Boyce, Welsh singer and comedian.
, P., Montgomery Montgomery, city, United States
Montgomery, city (1990 pop. 187,106), state capital and seat of Montgomery co., E central Ala., near the head of navigation on the Alabama River just below the confluence of the Coosa and Tallapoosa rivers, and in the rich
, C. H., & Tenopir. C. (2003). Library economic measures: Examples of the comparison of electronic & print journal collections and collection services. Library Trends, 51(3), 376-400.

Kyrillidou, M. (2002). From input and output measures to quality and outcome measures, or from the user in the life of the library to the library in the life of the user. Journal of Academic Librarianship, 28(1-2), 42-46.

Lakos, A. (1999). The missing ingredient
This article is about ingredients in general. There is also an American soul and R&B group called The Main Ingredient.

An ingredient is something that forms part of a mixture (in a general sense).
: Culture of assessment in libraries. Performance Measurement and Metrics metrics Managed care A popular term for standards by which the quality of a product, service, or outcome of a particular form of Pt management is evaluated. See TQM. , 1(1), 3-7. Retrieved March 20, 2003, from

Lance, K. C., Steffen, N. O., Logan Logan, city (1990 pop. 32,762), seat of Cache co., N Utah, on the Logan River; inc. 1859. It is the center of an irrigated dairy and farm area, with huge cheese plants, other food-processing facilities, and diverse manufactures. , R., Rodney Rodney may refer to:

  • Caesar Rodney (1728–1784), Signer of the U.S. Declaration of Independence and President of Delaware
  • Caesar A. Rodney (1772–1824), U.S.
, M. J., & Kaller, S. (2002). Counting on results: New tools for outcomes-based evaluation of public libraries. Aurora Aurora, cities, United States
Aurora (ərôr`ə, ô–).

1 City (1990 pop. 222,103), Adams and Arapahoe counties, N central Colo., a growing suburb on the east side of Denver; inc. 1903.
, CO: Bibliographical Center for Research The Bibliographical Center for Research, commonly known as BCR, is a "nonprofit, multistate library cooperative."[1] It was founded in 1935 and now includes "more than 8,000 member libraries in 42 states, Canada and Guam. . Retrieved January January: see month.  6, 2003, from CountingOnResults.htm.

McClure, C. R., & Bertot, J. C. (1998). Public library use in Pennsylvania Pennsylvania (pĕnsəlvā`nyə), one of the Middle Atlantic states of the United States. It is bordered by New Jersey, across the Delaware River (E), Delaware (SE), Maryland (S), West Virginia (SW), Ohio (W), and Lake Erie and New York : Identifying uses, benefits, and impacts. Harrisburg Harrisburg, city (1990 pop. 52,376), state capital and seat of Dauphin co., SE Pa., on the Susquehanna River; settled c.1710 by John Harris, who established a trading post and operated a ferry there; inc. 1791. , PA: Office of Commonwealth Libraries, Department of Education.

McClure, C. R., & Bertot, J. C. (Eds.). (2001). Evaluating networked information services: Techniques, policy, and issues. Medford, NJ: Information Today.

McClure, C. R., Fraser, B. T., Shim, W., & Bertot, J. C. (2002). Identifying and measuring library activities/services related to academic institutional outcomes. Washington, D.C.: Association of Research Libraries.

McClure, C. R., Lankes, R. D., Gross, M., & Choltco-Devlin, B. (2002). Statistics, measures, and quality standards for assessing digital reference library services: Guidelines and procedures. Syracuse Syracuse, city, Italy
Syracuse (sĭr`əkys, –kyz), Ital. Siracusa, city (1991 pop.
, NY: Information Institute of Syracuse.

Shim, W.J., & McClure, C. R. (2002). Improving database vendors' usage statistics reporting through collaboration Working together on a project. See collaborative software.  between libraries and vendors. College & Research Libraries, 63(6), 1-16.

Shim, W.J., McClure, C. R., Fraser, B. T., & Bertot, J. C. (2002). Data collection manual for academic and research library network statistics and performance measures. Washington, D.C.: Association of Research Libraries.

Smith, K. R. (2000). New roles and responsibilities for the university library: Advancing student learning through outcomes assessment. Washington, D.C.: Association of Research Libraries. Retrieved November November: see month.  8, 2002, from

Smith, M., & Rowland, G. (1997, May-June). To boldly go
  • For the Star Trek quotation from which this phrase is best known, see Where no man has gone before.
  • For the play-by-web turn-based strategy game, see To Boldly Go.

To Boldly Go (commonly known as TBG
: Searching for output measures for electronic services, Public Libraries, 168-172.

United Way of America United Way of America: see community chest. . (1996). Measuring program outcomes: A practical approach. Alexandria Alexandria, city, Egypt
Alexandria, Arabic Al Iskandariyah, city (1996 pop. 3,328,196), N Egypt, on the Mediterranean Sea. It is at the western extremity of the Nile River delta, situated on a narrow isthmus between the sea and Lake Mareotis (Maryut).
, VA: United Way of America.

Van House, N., Lynch, M.J., McClure, C. R., Zweizig, D. L., & Rodger Rodger is a surname, and may refer to:
  • Alan Rodger, Baron Rodger of Earlsferry (born 1944), Scottish judge
  • George Rodger (1908–1995), British photojournalist
  • N. A. M.
, E. J. (1987). Output measures for public libraries: A manual of standardized standardized

pertaining to data that have been submitted to standardization procedures.

standardized morbidity rate
see morbidity rate.

standardized mortality rate
see mortality rate.
 procedures. Chicago: American Library Association.

John Carlo Bertot, Associate Professor and Associate Director, and Charles Charles, archduke of Austria
Charles, 1771–1847, archduke of Austria; brother of Holy Roman Emperor Francis II. Despite his epilepsy, he was the ablest Austrian commander in the French Revolutionary and Napoleonic wars; however, he was handicapped by
 R. McClure, Francis Francis, French prince, duke of Alençon and Anjou
Francis, 1554–84, French prince, duke of Alençon and Anjou; youngest son of King Henry II of France and Catherine de' Medici.
 Eppes Eppes may refer to several American people:
  • Francis Eppes
  • Francis W. Eppes -- Intendant Mayor of Tallahassee, Florida.
  • John Wayles Eppes -- Democrat; U.S. Representative from Virginia; U.S.
 Professor and Director, Information Use Management Policy Institute, School of Information Studies, 242 Louis Louis, titular duke of Burgundy
Louis, 1682–1712, titular duke of Burgundy; grandson of King Louis XIV of France. He became heir to the throne on the death (1711) of his father, Louis the Great Dauphin.
 Shores Building, Florida State University Florida State University, at Tallahassee; coeducational; chartered 1851, opened 1857. Present name was adopted in 1947. Special research facilities include those in nuclear science and oceanography. , Tallahassee, FL 32306-2100

JOHN CARLO BERTOT is Associate Professor and Associate Director of the Information Use Management and Policy Institute in the School of Information Studies at Florida State University. He teaches courses in library technology planning, technology applications, and information and telecommunications Communicating information, including data, text, pictures, voice and video over long distance. See communications.  policy. Bertot has published extensively in the areas of library management, planning, and evaluation, with a particular emphasis on library Internet use and involvement. Most recently, he coauthored Statistics and Performance Measures for Public Library Networked Services through the American Library Association (2001), and coedited Evaluating Networked Information Services, an Information Today publication (2002). At present, Bertot is co-principal investigator for a national U.S. study sponsored by the Institute of Museum and Libraries to develop a national model for the collection and reporting of public library network statistics and performance measures as well as for the development of training modules to assist libraries in collecting, using, and reporting network-services data. Bertot also serves as a U.S. delegate A person who is appointed, authorized, delegated, or commissioned to act in the place of another. Transfer of authority from one to another. A person to whom affairs are committed by another.

A person elected or appointed to be a member of a representative assembly.
 on the International Standards Organization's (ISO (1) See ISO speed.

(2) (International Organization for Standardization, Geneva, Switzerland, An organization that sets international standards, founded in 1946. The U.S. member body is ANSI.
) Library Statistics committee, and is the convener con·vene  
v. con·vened, con·ven·ing, con·venes

To come together usually for an official or public purpose; assemble formally.
 of the ISO Library Performance Indicator committee. He is also a member of the National Information Standards Organization's (NISO (National Information Standards Organization, Baltimore, MD, A non-profit organization founded in 1939 that deals with bibliographic and related information standards. ) planning committee planning committee n (in local government) → comité m de planificación  for the revision of the Z39.7 Library Statistics standard.

CHARLES R. MCCLURE, who obtained his Ph.D. from Rutgers University Rutgers University, main campus at New Brunswick, N.J.; land-grant and state supported; coeducational except for Douglass College; chartered 1766 as Queen's College, opened 1771. Campuses and Facilities

Rutgers maintains three campuses.
, is the Francis Eppes Professor of Information Studies at the School of Information Studies, Florida State University. He is also the Director of the Information Use Management and Policy Institute at Florida State University. McClure was the co-principal investigator on the Association of Research Libraries' funded E-metrics project, 2000-01; he was the co-principal investigator on a study funded by the U.S. Institute for Museum and Library Services to develop and test measures for networked services, 2000-02; and he recently completed a study funded by OCLC OCLC - Online Computer Library Center  and a consortium of other library organizations that resulted in a manual to assess digital reference services. He is the coauthor co·au·thor or co-au·thor  
A collaborating or joint author.

tr.v. co·au·thored, co·au·thor·ing, co·au·thors
To be a collaborating or joint author of: "He and a colleague . . .
 of Statistics and Performance Measures for Public Library Networked Services (American Library Association, 2001); Evaluating Networked Information Services (Information Today, 2002); and Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services (Syracuse University Press Syracuse University Press, founded in 1943, is a university press that is part of Syracuse University. External link
  • Syracuse University Press
, 2002).
COPYRIGHT 2003 University of Illinois at Urbana-Champaign
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2003, Gale Group. All rights reserved. Gale Group is a Thomson Corporation Company.

 Reader Opinion




Article Details
Printer friendly Cite/link Email Feedback
Author:McClure, Charles R.
Publication:Library Trends
Date:Mar 22, 2003
Previous Article:Public library service to children and teens: a research agenda.
Next Article:Electronic publishing: research issues for academic librarians and users.

Related Articles
An emergency department perspective: outcomes.
Principles and a case study.
Where the environment fits in public health.
The Regulatory Dilemma.
Getting Real: Implementing General Education Assessment that Works.
World libraries on the information superhighway: Internet-based library services.
Reflection through the ID-PRISM: a teacher planning tool to transform classrooms into web-enhanced learning environments.
Lessons learned for the National Children's Study from the National Institute of Environmental Health Sciences/U.S. Environmental Protection Agency...

Terms of use | Copyright © 2014 Farlex, Inc. | Feedback | For webmasters