Outcomes assessment in the networked environment: research questions, issues, considerations, and moving forward.ABSTRACT
THIS ARTICLE IDENTIFIES A NUMBER OF RESEARCH TOPICS related broadly to outcomes assessment in a networked environment and discusses issues affecting these research topics. It also proposes a framework to relate traditional evaluation components and terminology to the networked environment and identifies a number of factors in the networked environment that affect outcomes and other assessment methods. The article suggests that outcomes assessment has the potential to complement other assessment techniques to better assist libraries and related information organizations enhance their decisions in the provision of information services See Information Systems. and resources. Given the increased rate at which libraries are using the networked environment to provide services and resources, however, much work remains before most libraries can implement outcomes assessment efforts successfully.
Organizations of many types in various operating environments In computing, an operating environment is the environment in which users run programs, whether in a command line interface, such as in MS-DOS or the Unix shell, or in a graphical user interface, such as in the Macintosh operating system. have used a variety of performance assessment activities for some time. Libraries are no different, having engaged in the use of output, service quality, performance indicator, balanced scorecard Balanced Scorecard
A performance metric used in strategic management to identify and improve various internal functions and their resulting external outcomes. The balanced scorecard attempts to measure and provide feedback to organizations in order to assist in implementing , and a number of other performance measurement techniques. At the core of these measurement activities are a number of basic questions:
* What resources are required to support the services that a library provides?
* What services and/or and/or
Used to indicate that either or both of the items connected by it are involved.
Usage Note: And/or is widely used in legal and business writing. resources is a library able to provide with its investments in library infrastructure (broadly defined as personnel, technology, collections, facilities, etc.)?
* Are the library's customers receiving value out of the community's investment in library services?
* What is the real and/or perceived per·ceive
tr.v. per·ceived, per·ceiv·ing, per·ceives
1. To become aware of directly through any of the senses, especially sight or hearing.
2. To achieve understanding of; apprehend. quality of library services?
* What are the impacts of library services and resources on the community that the library serves?
* What level of effort is required by library staff to implement measurement activities?
* In what ways does the library contribute to the overall mission, goals, and objectives of the community--or institutions within that community--the library serves?
The role and usefulness of outcome measures and outcomes assessment in answering these and related questions, although potentially rich, requires significant additional review, research, field testing, and development.
The purpose of this article is to:
* Identify selected research questions regarding outcomes assessment;
* Review a number of issues affecting outcomes assessment in general and outcomes assessment in a networked environment in particular; and
* Propose a framework to relate and better understand traditional evaluation components and terminology.
The article suggests that outcomes assessment has the potential to evolve to a point at which it will complement other assessment techniques that assist libraries and related information organizations make better decisions in the provision of information services and resources. There are, however, a number of research, methodological, and other issues that require consideration prior to reaching this potential. Given the library community's increased reliance on providing services and resources via the networked environment, much work lies ahead before libraries can engage successfully in outcomes assessment activities.
KEY RESEARCH QUESTIONS
Outcomes assessment is a relatively new activity to libraries, and as such, this is a very broad area for research. Indeed, as with any new area of scholarly endeavor, there are many more questions than answers. Some key research questions are:
* What are the prevailing models/approaches for library services/resources assessment?
* Are these models distinct or interrelated in·ter·re·late
tr. & intr.v. in·ter·re·lat·ed, in·ter·re·lat·ing, in·ter·re·lates
To place in or come into mutual relationship.
* To what degree do these models/approaches consider networked services/resources?
* How are "outcomes" considered in these various models?
* What do these models/approaches tell researchers and practitioners about use, uses, quality, impact, etc. of library services/resources?
* What outcomes assessment methods will be most useful and successful given the needs of a library, the resources available to the library for assessment, the assessment activity objectives, staff skills available, and other situational factors?
* What are the key variables to consider in the development of such models?
This article provides a foundation upon which to begin addressing these questions. The authors do not claim, however, that these are the only research questions that require attention regarding library outcomes assessment activities. These questions are, though, essential to further the development of outcomes assessment approaches that provide libraries with techniques that identify the impacts and benefits of library services/resources on the communities that they serve. The remainder of the article presents the issues and considerations regarding these research questions as a means of moving the research agenda forward.
The Current Context and the Need for Answers
The environment in which libraries find themselves at present creates a situation in which answers to the above posed research questions are essential. Libraries of all types, but public and academic libraries in particular, find that they are considered increasingly as part of the larger organizational structure This article has no lead section.
To comply with Wikipedia's lead section guidelines, one should be written. in which they reside--university departments or local government agencies. As such, library funding and continued well-being is no longer distinct from that of any other campus or local agency. Because of this situation, libraries are being asked to:
* Articulate articulate /ar·tic·u·late/ (ahr-tik´u-lat)
1. to pronounce clearly and distinctly.
2. to make speech sounds by manipulation of the vocal organs.
3. to express in coherent verbal form.
4. the importance of and need for their services and resources;
* Identify the use and uses of their services and resources; and
* Establish the value, impacts, and benefits that the community receives from the library services and resources.
Given the above circumstances CIRCUMSTANCES, evidence. The particulars which accompany a fact.
2. The facts proved are either possible or impossible, ordinary and probable, or extraordinary and improbable, recent or ancient; they may have happened near us, or afar off; they are public or , it behooves the library practitioner and research communities to have assessment tools and approaches that enable libraries to articulate their contribution to the well-being of the communities that they serve.
In addition to this management context, there is the evolving technology context in which libraries operate. To state the obvious, library services and resources rely increasingly on:
* Technologies that continue to change and evolve at an ever-intensifying pace. On the one hand, these technology changes enable new services and resources that allow libraries to better meet the service and resource needs of their customers. On the other hand, this continual cycle of adoption, change, and new and/or enhanced services Enhanced service is service offered over commercial carrier transmission facilities used in interstate communications, that employs computer processing applications that act on the format, content, code, protocol, or similar aspects of the subscriber's transmitted information; and resources creates a number of challenges--including assessment challenges--for libraries.
* Leased and network-based resources and situations that remain outside the direct control of libraries (see Figure 1). For example, libraries continue to grow their subscriptions to online databases and resources (e.g., e-books). Moreover, customer access to those services and resources can occur through a number of nonlibrary venues--each of which has implications for what libraries and vendors can collect and report regarding those user-initiated activities. As such, libraries are in a situation in which they do not control the services themselves (libraries are merely subscribers to resources such as EbscoHost, ScienceDirect ScienceDirect is one of the largest online collections of published scientific research in the world. Produced by Elsevier it contains over 8.5 million articles from over 2000 journals, including titles such as The Lancet, Cell and Tetrahedron , and NetLibrary), do not control the path through which customers gain access to the service/resource (remote access is beyond the scope of the library), and do not control the customer service/resource use and interaction data. All of this has a direct impact on the assessment activities in which a library can engage, the findings derived from such activities, and the types of questions that libraries can ask users regarding their use and/or assessment of the services/resources accessed (Shim A small piece of software that is added to an existing system program or protocol in order to provide some enhancement.
(jargon, memory management) shim - A small piece of data inserted in order to achieve a desired memory alignment or other addressing property. & McClure Mc·Clure , Samuel Sidney 1857-1949.
Irish-born American editor and publisher who founded McClure's Magazine (1893), an influential muckraking periodical. , 2002).
[FIGURE 1 OMITTED]
Added to this mix is the impression that some outside the library community regard the information content as available without charge to Internet users Internet user n → internauta m/f
Internet user Internet n → internaute m/f . As libraries struggle to identify, define, and articulate the impacts derived by customers through the use of library services and resources, libraries must educate the communities that they serve regarding the issues associated with networked-based information content.
There are a number of evaluation methodologies that exist--for example, return on investment, quality assessment, outcomes assessment, outputs, best practices--to assist libraries in identifying and communicating their service/resource impacts on their communities. There are also a number of situational factors that affect a library's or a researcher's ability to identify, study, and present results of that work. This article describes a selected number of those methodologies, explores the relationship between them, and presents issues associated with methods and what they can tell us about library services and resources.
A first step in addressing the research questions posed earlier is defining and understanding an "outcome." There is no single concise definition of what an "outcome" means in the context of library service. During the past decade, a number of writers and researchers have proposed definitions that Table 1 summarizes. A review of various definitions, however, does yield a number of common elements. In general, outcomes:
* Include the notion of an impact, benefit, difference, or change in a user, group, or institution based on the use of or involvement with a library service or resource;
* Are predetermined pre·de·ter·mine
v. pre·de·ter·mined, pre·de·ter·min·ing, pre·de·ter·mines
1. To determine, decide, or establish in advance: based on a service/resource planning process in which the library engages to produce desired service/resource outcomes through the setting of service/resource goals and objectives; and
* Involve measuring and demonstrating the extent to which library services/resources meet the anticipated outcomes determined by the library or imposed by the community the library serves (e.g., academic institution, county, city).
These definitions broadly, therefore, consider outcomes assessment to be a proactive endeavor on the part of the library in which there is an a priori a priori
In epistemology, knowledge that is independent of all particular experiences, as opposed to a posteriori (or empirical) knowledge, which derives from experience. determination of the library service/resource preferred outcomes.
In general, these definitions assume that there is a larger context in which libraries reside that provides the basis for the library service/resource outcomes. In the case of academic libraries, for example, the desired outcomes might take the form of accreditation accreditation,
n a process of formal recognition of a school or institution attesting to the required ability and performance in an area of education, training, or practice. standards set forth by accreditation bodies; accountability The traceability of actions performed on a system to a specific system entity (user, process, device). For example, the use of unique user identification and authentication supports accountability; the use of shared user IDs and passwords destroys accountability. measures imposed by university administrators/ boards; or state government-imposed higher education higher education
Study beyond the level of secondary education. Institutions of higher education include not only colleges and universities but also professional schools in such fields as law, theology, medicine, business, music, and art. outcomes. Finally, these definitions assume that libraries are able to measure their service/resource outcomes with reliable and valid instruments and demonstrate subsequently that library service/resource outcomes contribute to the mission, goals, and objectives of the community that the library serves.
Over time, library researchers, managers, and others have developed a number of models to describe and relate inputs, outputs, service quality, outcomes, impacts, etc. Kyrillidou (2002) summarizes some of these approaches. There is no lack of proposed models to describe these evaluation components (e.g., Hernon, 2002, p. 55; Cook & Heath heath, tract of open land
heath, tract of open land characterized by a few scattered trees, abundant moss cover, and numerous low shrubs, principally of the heath family (see heath, in botany). , 2001, p. 580; Dresang & Gross, 2001, p. 28). While some might suggest that the plethora plethora /pleth·o·ra/ (pleth´ah-rah)
1. an excess of blood.
2. by extension, a red florid complexion.pletho´ric
1. of views, models, and definitions describe a healthy intellectual development in library/information services and resources evaluation, others--namely many practitioners--may not agree. Part of the issue is that each of these views, models, and evaluation approaches Evaluation approaches are conceptually distinct ways of thinking about, designing and conducting evaluation efforts. Many of the evaluation approaches in use today make truly unique contributions to solving important problems, while others refine existing approaches in some way. is presented individually without any review or consideration of the relationship between the key aspects of these models or perspectives.
Outcomes assessment, although not new to other organizations and sectors, is relatively new to libraries and focuses on determining the impact of a library's services and/or resources on its customers. In the broadest sense, outcomes assessment focuses on the extent to which a library's services and/or resources made a difference in the life of the library's individual, group, or institutional users (see Table 2). Writers and researchers, however, do not agree on what it means to measure outcomes.
Kyrillidou (2002) presents three models that depict de·pict
tr.v. de·pict·ed, de·pict·ing, de·picts
1. To represent in a picture or sculpture.
2. To represent in words; describe. See Synonyms at represent. the possible relationship between inputs, outputs, outcomes, and service quality. These models include a linear model, cyclic model In the 1930s, theoretical physicists, most notably Einstein, considered the possibility of a cyclic model for the universe as an (everlasting) alternative to the Big Bang. However, work by Richard C. , and spiral spiral /spi·ral/ (spi´ral)
1. helical; winding like the thread of a screw.
2. helix; a winding structure. swirl (p. 44). Without empirical testing, Kyrillidou finds problems with all the models presented save the spiral swirl as it apparently can depict motion and a more flexible and dynamic process that intertwines inputs, outputs, outcomes, and service quality. Ultimately, Kyrillidou concludes that service quality is an outcome of library services and resources that exist through a library resource input and output process.
Hernon (2002) reaches a different conclusion, stating that service quality and satisfaction and outcomes assessment "truly stand out as central assessment concepts for librarianship li·brar·i·an
1. A person who is a specialist in library work.
2. A person who is responsible for a collection of specialized or technical information or materials, such as musical scores or computer documentation. . Inputs and outputs are important to institutions and accrediting bodies as measure of efficiency and crude measure of effectiveness" (p. 55). In Hernon's view, therefore, there is little to no relationship between inputs, outputs, service quality, and outcomes assessment--and libraries should focus on service quality and outcomes assessment techniques. King et al. (2002) take a different approach by linking types of measurement perspectives (library, user, organization, etc.) with specific types of measures--one of which is outcomes. The approaches shown in Table 2 are illustrative il·lus·tra·tive
Acting or serving as an illustration.
Adj. 1. of recent writings, though historically many writers have offered models and approaches related to inputs, measures, service quality, and outcomes.
Table 2, based on research by the authors and others, summarizes the view of the outcomes assessment environment of public and academic libraries from selected perspectives. Each of the described approaches has strengths and weaknesses; each makes implicit assumptions about outcomes and the relationships between outcomes and other evaluation factors; and most do not address a range of factors and considerations discussed later in this article. In short, it is not possible to address meaningfully the research questions posed initially in this article with such different views of outcomes and the relationship of outcomes to other evaluation approaches.
The authors of this article present a view summarized in Figure 2. This model, based on a number of large-scale large-scale
1. Large in scope or extent.
2. Drawn or made large to show detail.
1. wide-ranging or extensive
2. field-tested and empirical research Noun 1. empirical research - an empirical search for knowledge
inquiry, research, enquiry - a search for knowledge; "their pottery deserves more research than it has received" efforts, posits that inputs and outputs form the basis for service quality and outcomes assessment activities (Bertot, McClure, & Davis, 2002; Shim et al., 2002; Bertot, McClure, & Ryan Ryan may refer to: Places
put differently , one cannot have outcomes measures without measures of outputs--libraries need to know what investments (inputs) produce what services (outputs) in order to determine the perceived quality (quality assessment) and impacts (outcomes) of those services/resources.
[FIGURE 2 OMITTED]
An Outcomes Framework--More than One Type of Outcome
Much of the outcomes literature presents outcomes as monolithic--that is, there is a set of generic outcomes that fit various organizational settings (IMLS IMLS Institute of Museum and Library Services
IMLS Institute for Museum and Library Services (US)
IMLS Institute of Medical Laboratory Sciences , 2000). However, it is important to recognize that there are many types of outcomes, such as research and learning outcomes (Hernon & Dugan |
Dugan is a name of Irish heritage. Other common modern variants are Duggan, Dougan, Doughan, Doogan, and Duggin.
Fraser, chief river of British Columbia, Canada, c.850 mi (1,370 km) long. It rises in the Rocky Mts., at Yellowhead Pass, near the British Columbia–Alta. line and flows northwest through the Rocky Mt. & McClure, 2002). Developing a framework for outcomes assessment requires a complex analysis that encompasses the operating environment of the library, the impact of situational factors on library services and resources outcomes, and the reality that it is not always possible for libraries to anticipate and/or predict the outcomes of their services/resources on users.
Different types of outcomes have been suggested by Lance et al. (2002), Hernon and Dugan (2002), Bertot (2001), and Fraser and McClure (2002)--to name a few. As a result, developing a framework for outcomes assessment requires a complex analysis that considers a number of outcome types:
* Economic. Economic outcomes would include the impact of library services and resources on the ability of library users to prosper financially, seek employment successfully, or develop and sustain a small business in the community;
* Learning. Learning outcomes would include the impact of library services and resources on the ability of library users to engage in lifelong learning Lifelong learning is the concept that "It's never too soon or too late for learning", a philosophy that has taken root in a whole host of different organisations. Lifelong learning is attitudinal; that one can and should be open to new ideas, decisions, skills or behaviors. , interact with and engage a number of information resources (1) The data and information assets of an organization, department or unit. See data administration.
(2) Another name for the Information Systems (IS) or Information Technology (IT) department. See IT. , develop information literacy Several conceptions and definitions of information literacy have become prevalent. For example, one conception defines information literacy in terms of a set of competencies that an informed citizen of an information society ought to possess to participate intelligently and skills, develop technology skills, become literate, or develop an analytic an·a·lyt·ic or an·a·lyt·i·cal
1. Of or relating to analysis or analytics.
2. Expert in or using analysis, especially one who thinks in a logical manner.
3. Psychoanalytic. ability to assess the validity and reliability of information sources;
* Research. Such outcomes might include the impacts of library services and resources on the research process of faculty and students in a university--e.g., assistance in proposal writing, grant receipt, and publication;
* Information Exchange. Information exchange outcomes would include the impact of library services and resources on the ability of library users to interact with government, exchange information with distant family, or receive information regarding countries of interest (e.g., foreign newspapers and other sources);
* Cultural. Cultural outcomes would include the impact of library services and resources on the ability of library users to develop an appreciation for fine arts, history, music, diversity, or other societal so·ci·e·tal
Of or relating to the structure, organization, or functioning of society.
Adj. aspects; and
* Community. Some library outcomes affect the local community, be that an academic setting, a city or town, or a virtual community. Such outcomes could affect the overall quality of life for members of the community, attitudes of community members toward services, or even the political landscape of the community.
These categories are not mutually exclusive Adj. 1. mutually exclusive - unable to be both true at the same time
incompatible - not compatible; "incompatible personalities"; "incompatible colors" but rather are intended to be illustrative of types of outcomes. It is possible to identify other types of outcomes, as well as to expand the nature of the outcomes categories presented above. The key point is, however, that outcomes can be many and varied--and produced by the same service or resource (e.g., public access Internet Internet
Publicly accessible computer network connecting many smaller networks from around the world. It grew out of a U.S. Defense Department program called ARPANET (Advanced Research Projects Agency Network), established in 1969 with connections between computers at the workstations, online databases).
As identified in Table 3, there are two additional aspects of outcomes that researchers and practitioners need to consider: the level of outcome applicability and the time dimension of the outcome itself (discussed below). It is important to note that outcome types may apply at different levels (illustrated by the columns in Table 3). For example, a learning outcome may apply to the:
* Individual user of the library's services/resources by gaining the ability to read;
* Library by having a now-literate customer consume other library services/resources and derive additional impacts/benefits;
* Community by now having a literate member who can seek and expand his/her employability; and
* State and nation through a more economically solvent solvent, constituent of a solution that acts as a dissolving agent. In solutions of solids or gases in a liquid, the liquid is the solvent. In all other solutions (i.e. individual who contributes more to the economy through higher wages.
As such, outcomes can have multiple impacts for the individual user, library, library institution (e.g., university), system (i.e., state university system, public library system as a whole), state, and nation.
According to according to
1. As stated or indicated by; on the authority of: according to historians.
2. In keeping with: according to instructions.
3. some authors, outcomes assessment implies that outcomes are predefined through a planning process in which libraries engage (Hernon & Dugan, 2002; Fraser & McClure, 2002). In this view, outcomes are established a priori through service and resource goals and objectives developed through a planning process; there is some type of pretest pre·test
a. A preliminary test administered to determine a student's baseline knowledge or preparedness for an educational experience or course of study.
b. A test taken for practice.
2. in which libraries ascertain the extent to which their services/resources meet these goals and objectives according to users; and then, after user exposure to the new and/or modified library services/resources, the library measures the impact (outcome) of the services/resources through a posttest post·test
A test given after a lesson or a period of instruction to determine what the students have learned. . Again, this is an oversimplification o·ver·sim·pli·fy
v. o·ver·sim·pli·fied, o·ver·sim·pli·fy·ing, o·ver·sim·pli·fies
To simplify to the point of causing misrepresentation, misconception, or error.
v.intr. of the outcomes environment. Indeed, it is possible to have outcomes that are on different time dimensions as the rows in Table 3 demonstrate. Outcomes can be:
* Anticipated. These are the outcomes for which the library plans and by which the library intends to measure its success/failure in goals and objectives attainment. The library expects to achieve certain outcomes through its services/resources and then seeks to ascertain the extent to which its services/resources achieved the anticipated outcomes. It is important to note that anticipated outcomes can be generated by the library or imposed externally by funding agencies (e.g., the Institute of Museum and Library Services The Institute of Museum and Library Services is an independent agency of the United States federal government. It is the main source of federal support for libraries and museums within the United States. ) or governing gov·ern
v. gov·erned, gov·ern·ing, gov·erns
1. To make and administer the public policy and affairs of; exercise sovereign authority in.
2. authorities (e.g., university boards and provosts, accreditation bodies, city council, etc.). Most outcomes work to date focuses on this type of outcome assessment model.
* Emergent emergent /emer·gent/ (e-mer´jent)
1. coming out from a cavity or other part.
2. pertaining to an emergency.
1. coming out from a cavity or other part.
2. coming on suddenly. . These are outcomes that emerge through the service/resource planning and implementation process. Such outcomes are not the immediate focus of the service/resource goals and objectives--either library or externally imposed. However, as the library develops its service/ resource plans, these additional outcomes become apparent and are incorporated into the assessment process.
* Unanticipated. Once a service/resource is in operation, there are those outcomes that derive from actual service/resource use or interaction and can be ones that neither the library nor others predicted--nor planned to assess. For example, in the early years of public library Internet connectivity, many public librarians This is a list of people who have practised as a librarian and are well-known, either for their contributions to the library profession or primarily in some other field. anticipated that the primary outcome of Internet connectivity in their libraries would be for information seeking Information seeking is the process or activity of attempting to obtain information in both human and technological contexts. Information seeking is related to, but yet different from, information retrieval (IR). , and quickly discovered that the primary outcome was communications--the ability of users to stay in touch with family and countries (McClure & Bertot, 1998). This outcome is still prevalent prevalent
widespread occurrence. in public libraries, as demonstrated recently by Lance et al. (2002). By the same token In programming, a string of characters. For example, in the C expression #define MAXAMOUNT 50000, MAXAMOUNT is the token. See also token passing and authentication token.
1. , universities enhanced their Internet connectivity in their goal to become "wired campuses." Of the many anticipated outcomes
associated with robust connectivity, one unintended outcome was the substantial use of university resources by students to access online music services (some later deemed illegal). A number of universities now block access to such services from their campuses to preserve their connectivity resources, as well as to limit the universities from liability due to various copyright infringements.
It is clear that outcomes are not always anticipated either by librarians or others with expectations for library service/resource use. By focusing on the anticipated outcomes, librarians (and others) ignore significant emergent and unanticipated outcomes--both positive and negative. Any outcomes assessment strategy, therefore, needs to incorporate the ability to discover and articulate outcomes that were not the intended focus of the services/resources--something not likely in a pre- pre- word element [L.], before (in time or space).
1. Earlier; before; prior to: prenatal.
2. or posttest quasi-experimental methodology.
Outcomes Assessment Methodology Issues
Embedded Inserted into. See embedded system. within the discussion of the types of outcomes identified above is the notion of gap identification. In an ideal situation, a library anticipates a set of outcomes from its services/resources, and the library's user community confirms those anticipated outcomes. It is more likely the case, however, that outcomes reside within a zone of tolerance in which library intended outcomes and user assessment of those outcomes are within an acceptable range--or minimal gap. There may be multiple gaps or zones of tolerance--the library, the university, the user--all of these constituencies comprise the outcomes space within which the library's services/resources reside. There is a need to explore further the notion of gap analysis in outcomes assessment activities such as is being done with service quality research activities (Cook et al., 2001).
Another critical factor in outcomes assessment is the ability to isolate isolate /iso·late/ (i´sah-lat)
1. to separate from others.
2. a group of individuals prevented by geographic, genetic, ecologic, social, or artificial barriers from interbreeding with others of their kind. the actual impact of library services/resources on the user community. As an example, we use one of Smith's (2000) institutional learning outcomes for university libraries, Develop Attitudes of Openness, Flexibility, Curiosity Curiosity
so assured of wife’s fidelity, asks friend to try to corrupt her; friend is successful. [Span. Lit.: Don Quixote]
Cupid and Psyche
her inquisitiveness almost drives him away forever. [Gk. Myth. , Creativity, and an Appreciation of the Value of a Broad Perspective. Let us create a formula based on general linear model (GLM GLM Global Language Monitor
GLM Global Marine (stock symbol)
GLM Graduated Length Method (ski instruction)
GLM Good Looking Mom (used in pediatric practices)
GLM God Loves Me ) analysis techniques for which one might use analysis of variance The discrepancy between what a party to a lawsuit alleges will be proved in pleadings and what the party actually proves at trial.
In Zoning law, an official permit to use property in a manner that departs from the way in which other property in the same locality (ANOVA anova
see analysis of variance.
ANOVA Analysis of variance, see there ) or multiple regression Multiple regression
The estimated relationship between a dependent variable and more than one explanatory variable. to analyze an·a·lyze
1. To examine methodically by separating into parts and studying their interrelations.
2. To separate a chemical substance into its constituent elements to determine their nature or proportions.
Dependent Variable = Students develop attitudes of openness, flexibility, curiosity, creativity, and an appreciation of the value of a broad perspective
Independent Variable (s) = Particular library services/resources + courses + student extracurricular activities + ... +?
In quantitative terms, researchers as well as librarians and university officials will want to know what library services/resources contribute, in what way, and how much, to the student body's development of attitudes of openness, flexibility, curiosity, creativity, and an appreciation of the value of a broad perspective. In other words, the assessors of program outcomes will want to know which library services/resources contribute to the attainment of the outcome, as well as by how much, and in what way(s).
Needless to say, it is near impossible to quantify Quantify - A performance analysis tool from Pure Software. , much less isolate, a university library's contribution toward the outcome identified above. Simply put, student "attitudes" are formed by any number of campus activities in which students might engage--from coursework coursework
work done by a student and assessed as part of an educational course
Noun 1. coursework - work assigned to and done by a student during a course of study; usually it is evaluated as part of the student's to mentoring by faculty to clubs to which the student may belong, etc. Library services/resources do not exist in a vacuum and measurement techniques will need to take these external and likely contributing factors into account. This is particularly the case when outcomes are presented as an institutional (i.e., university) objective but measurement occurs locally (i.e., at the library). To ignore the other university activities that likely contribute to the outcome would yield an incomplete picture of library contributions to outcome attainment at best and distort the library's contribution to outcomes attainment at worst.
THE COMPLEXITY OF THE LIBRARY SERVICE ENVIRONMENT
Libraries now reside in a complex service environment--one that requires that they provide traditional services and resources such as a physical space, print material, and face-to-face (jargon, chat) face-to-face - (F2F, IRL) Used to describe personal interaction in real life as opposed to via some digital or electronic communications medium. reference, as well as network-based services such as Web-based collections, online databases, and virtual reference (Bertot, 2000; McClure, Fraser et al., 2002). Libraries are, therefore, providing more services and resources through multiple delivery methods. Moreover, while these traditional and network-based services are perhaps related and similar in function, they differ vastly in a number of significant ways such as the infrastructure required to deliver the services, the ways in which users or user communities access the services, the skills and knowledge required by uses or user communities to use the services, the reach and range of the services, the ways in which libraries manage such services, and the skills required within the library to deliver and access such services.
The networked environment is an intricate series of networks that interconnect (1) To attach one device to another.
(2) A physical port (plug, socket) or wireless port (transmitter, receiver) used to attach one device to another. in such a way that users never completely know the route they took to retrieve To call up data that has been stored in a computer system. When a user queries a database, the data are retrieved into the computer first and then transmitted to the screen.
(language) Retrieve the information or resource that they sought. As Smith and Rowland Row·land , F(rank) Sherwood Born 1927.
American chemist who shared a 1995 Nobel Prize for his work on the chemical processes involved in the formation and decomposition of ozone. (1997, p. 170) note, local users can access local resources, remote users can access local resources, and local users can access remote resources. Such a multifaceted mul·ti·fac·et·ed
Having many facets or aspects. See Synonyms at versatile.
Adj. 1. multifaceted - having many aspects; "a many-sided subject"; "a multifaceted undertaking"; "multifarious interests"; "the multifarious environment--one in which the library is the resource or the library is the resource gateway for local or remote users--creates a complex environment for measurement. Data collection and research activities, therefore, need to reflect this complexity and multidimensionality of electronic networks. They also need to recognize the need for collecting a range of data for use at the organizational (library), institutional (university, county government), state, or national levels.
The implications for outcomes assessment due to this environment are numerous. They include, but are not limited to:
* Many library network-based services are not truly the services of that library. For examples, libraries license the use of such vendor-based services as EbscoHost and ScienceDirect. As such, they do not own, but rather lease, such "collections." Therefore, any measurement activities that assess the outcomes of library services need to take into account that libraries do not have direct control over such services. Thus, how best to "count" basic networked service interactions is complex (Shim & McClure, 2002).
* Users can and do access network-based services from numerous locations using a wide range of technology and telecommunications services In telecommunication, the term telecommunications service has the following meanings:
1. Any service provided by a telecommunication provider.
2. . Each of these variations in access can and does affect the experience of the user--and these variations are beyond the purview The part of a statute or a law that delineates its purpose and scope.
Purview refers to the enacting part of a statute. It generally begins with the words be it enacted and continues as far as the repealing clause. of the library. Indeed, some library users do not access the library physically.
* The type of interaction that users can have with a library's network-based services and resources versus a library's physical services and resources can be quite different. Virtual users do not "browse (1) To view the contents of a file or a group of files. Browser programs generally let you view data by scrolling through the documents or databases. In a database program, the browse mode often lets you edit the data. See Web browser. the stacks" as do physical users. Moreover, not all of a library's collection is available in digital format; thus, virtual users are often limited to certain types of resources and services. In some cases, a service or resource is only available in electronic format. Another service that libraries provide increasingly is user training (e.g., use of technology, use of databases, other). Online tutorials differ greatly from in-person adj. 1. undertaken by an individual in person; as, an in-person appearance s>.
Adj. 1. in-person - an appearance carried out personally in someone else's physical presence; "he carried out the negotiations in person"; "a training in a library lab setting.
These issues suggest that research activities cannot regard all users as the same, nor can they treat all library services and/or service delivery modes as the same.
CONSIDERATIONS AND CONCERNS
As identified earlier in this article, the discussion of outcomes and outcomes measurement in libraries and related information organizations is in its infancy infancy, stage of human development lasting from birth to approximately two years of age. The hallmarks of infancy are physical growth, motor development, vocal development, and cognitive and social development. . There are more questions about outcomes and outcomes assessment than there are adequate answers at this point. Indeed, it may be that at some point in the future, researchers will determine that outcomes and outcomes assessment have little utility in library organizational and services evaluation because of problems of implementations--especially in a networked environment. The research questions posed in the beginning of this article only scratch the surface of those that require attention. As the research and library communities move forward in studies related to outcomes and outcomes assessment, there is a need to make explicit some assumptions, considerations, and concerns.
Need for Empirical Research
At present, there is no lack of opinions and views related to outcomes and outcomes assessment. But, there is a lack of empirical research related to outcomes, outcomes assessment, and conceptual frameworks For the concept in aesthetics and art criticism, see .
A conceptual framework is used in research to outline possible courses of action or to present a preferred approach to a system analysis project. that would help practitioners to understand better the basis for, application of, and use of outcomes assessment. One empirical effort was the Association of Research Libraries' sponsored E-metrics study (Association of Research Libraries, 2002). One aspect of the E-Metrics project explored the role of the academic library in its contribution to achieving institutional outcomes. Findings from the E-Metrics study suggest (Fraser and McClure, 2002):
* The inputs-outputs model for library assessment (which produced the statistics and measures in the report Measures and Statistics for Research Library Networked Services: Procedures'and Issues) may not be linked easily to demonstrating the library's role in accomplishing institutional outcomes.
* Many of the library's activities, resources, and services are combined with other institutional activities, resources, and services in such a way that parsing See parse.
parsing - parser out only the library's contribution to institutional outcomes is extremely difficult.
* The process by which libraries are involved in the identification of and agreement to these institutional outcomes is not clear.
* There is widespread confusion as to what an "institutional outcome" is and how such outcomes "fit" into traditional assessment procedures.
* Increasingly, academic accreditation agencies are considering the use and appropriateness of "institutional outcomes" as a means to assess the degree to which the organization determines what end products should result from organizational activities.
* The issue of nonexistent non·ex·is·tence
1. The condition of not existing.
2. Something that does not exist.
non or inconsistent Reciprocally contradictory or repugnant.
Things are said to be inconsistent when they are contrary to each other to the extent that one implies the negation of the other. incomparable (mathematics) incomparable - Two elements a, b of a set are incomparable under some relation <= if neither a <= b, nor b <= a. usage statistics provided by external information content providers (vendors) is a major stumbling block stum·bling block
An obstacle or impediment.
any obstacle that prevents something from taking place or progressing
Noun 1. for libraries to gauge rapidly increasing use of electronic resources by research library users, thus making it difficult to use the data as sources for establishing library outcomes.
To a large degree, the E-metrics study found that academic libraries are not well prepared to demonstrate the extent to which they contribute to the organization's accomplishment of institutional outcomes. Moreover, libraries are looking at outcomes at the library level when, in fact, they reside within an institutional context. Nonetheless, there appears to be considerable interest in identifying and measuring the degree to which the academic library does contribute to accomplishing such institutional outcomes.
One of the products from the E-metrics study was a proposal for additional research in the area of outcomes, "Identifying and Measuring Library Activities/Services Related to Academic Institutional Outcomes" (McClure et al., 2002). The proposal outlined a conceptual framework for understanding outcomes assessment that has yet to be tested. In fact, there are few efforts to propose conceptual frameworks (such as that offered in Figure 2 earlier in this article) to help researchers lodge such research in the broader frame of evaluation theory. Thus, empirical research projects are necessary to understand better and describe outcomes and outcomes assessment--regardless of the type of outcomes being investigated.
Need for Multimethod Approaches
From the issues identified and discussed in this article, it is clear that research endeavors cannot treat all users as the same, all delivery of library services in the networked environment as the same, nor should they lump generically library services for assessment purposes. The differences between online and physical services/resources are real and not comparable (Bertot, McClure, & Davis, 2002; Shim et al., 2002; Bertot, 2001). Research efforts that do not separate network-based library services and resources from physical services/resources--such as work by Hernon and Dugan (2002) and Hernon and Whitman Whitman, town (1990 pop. 13,240), Plymouth co., SE Mass., S of Boston; settled c.1670, set off from Abington and inc. 1875. It is an industrial town that manufactures shoes, plastics, foundry products, and textile machinery. The Toll House (1709) is restored. (2001)--miss substantive Substantive may refer to:
While there is a need to focus much research on outcomes, such research should not occur at the exclusion of promising work in other areas of assessment. For example, there is a range of other efforts currently underway to assist the library community in the assessment of services and resources. These other efforts focus on:
* Service discrepancies among different stakeholder stakeholder n. a person having in his/her possession (holding) money or property in which he/she has no interest, right or title, awaiting the outcome of a dispute between two or more claimants to the money or property. groups (e.g., LibQUAL+);
* Inputs analysis;
* Outputs analysis;
* Cost/benefit analysis;
* Performance measures (that can combine inputs and outputs); and
* Quality standards.
The issue is not which of these, or others, is the best approach for libraries. The issue is better stated as: Given the needs of the library, the resources available to the library for assessment, the assessment activity objectives, and the staff skills available--what methods for assessment will be most useful and successful?
Oftentimes of·ten·times also oft·times
Adv. 1. oftentimes - many times at short intervals; "we often met over a cup of coffee"
frequently, oft, often, ofttimes , a combination of methods is best as each method has its own strengths and weaknesses. For example the recently published manual Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services: Guidelines guidelines,
n.pl a set of standards, criteria, or specifications to be used or followed in the performance of certain tasks. and Procedures (McClure et al., 2002) offers some thirty-five performance measures related to the provision of digital reference services Digital reference is a service by which library reference service is conducted online, and the reference transaction is a computer-mediated communication.
The word "reference" in this context refers to the task of providing assistance to library users in finding information, as well as six suggested areas for quality standards. These measures and quality standards are a first comprehensive effort for assessing digital reference services. Inclusion of cost-benefit analysis cost-benefit analysis
In governmental planning and budgeting, the attempt to measure the social benefits of a proposed project in monetary terms and compare them with its costs. , service quality approaches, and outcomes assessment could also be incorporated with the approach described in the manual. As methods are integrated and expanded, however, the cost and level of effort required to conduct the assessment increases.
Multimethod approaches are also likely to be most useful given the range of situational factors that affect individual organizations and libraries. The type of assessment required for a small rural public library that is a member of a large regional cooperative with few staff and resources is likely to vary considerably from a large academic library that purchases an extensive collection of electronic resources, has talented technology staff, and provides customized networked services to its academic community.
In the experience of these authors, one evaluation method does not fit all types of libraries. The promotion of one particular method by some at the expense of all other methods does not reflect the complexity of situational factors as they relate to assessment in a library context. Nor might a single approach provide reliable and valid research results in all library organizational settings--there is a need for flexibility in methods and execution for libraries to engage successfully in evaluation activities.
Impact of the Networked Environment
As discussed earlier in this article, assessment of library services and resources--regardless of method--needs increasingly to consider the role and impact of electronic services and resources. Assessment of electronic services and resources, especially in a networked environment, involves a very different set of circumstances than those in the print and physical environment (McClure & Bertot, 2001). Assessment in a networked environment raises the following unique issues:
* The technology for the delivery of these services and resources is constantly changing, which affects what services/resources a library can deliver and how such delivery might occur. This also affects the methods, approaches, and software that enable the "tracking" of such services/resources;
* There is a need to have comparative (and valid) measures for traditional versus electronic-based services and resources, e.g., is a physical visit to the library (turnstile count) comparable to a visit to the library's Web site?
* Oftentimes, libraries contract (license) for networked services or do not control their own technology infrastructure (e.g., servers), thus they cannot easily obtain transactional data describing services and resource provision;
* The increased reliance on consortia and other group arrangements for the purchase and delivery of electronic services and resources blurs actual costs for these services and resources as well as how these services and resources are being used;
* Related to the above, leased services/resources are not truly the services/resources of the library. Rather, the library acts as a gateway to such services/resources. Outcomes assessment activities need to consider the most appropriate strategies for evaluating the outcomes of services/resources that are technically not the library's; and
* Librarian (1) A person who works in the data library and keeps track of the tapes and disks that are stored and logged out for use. Also known as a "file librarian" or "media librarian." See data library.
(2) See CA-Librarian. skills necessary to conduct outcomes and other types of assessment in the networked environment are significant and require constant enhancement given the changing nature of that environment.
Ease of Use
One major concern regarding the development of outcomes assessment methods is the ability of libraries to implement the recommended approaches, data collection activities, interpret the results, and use the findings to inform library resource/service planning efforts as well as external library stakeholders Stakeholders
All parties that have an interest, financial or otherwise, in a firm-stockholders, creditors, bondholders, employees, customers, management, the community, and the government. . Thus, it is imperative that researchers engaged in the development of outcomes assessment tools consider the degree to which the methods are practical and feasible for day-to-day day-to-day
1. Occurring on a routine or daily basis: the day-to-day movements of the stock market.
2. use in libraries and other information organizations. As this article indicates, however, the ability of researchers to meet this ease-of-use burden remains unclear as outcomes assessment requires complex indicators and methods.
To some degree, outcomes assessment is a theory in search of a practice. From the earliest days of the development of various output measurement manuals in the early 1980s, many practitioners remain unconvinced that the amount of time and resources necessary for conducting assessments--of any type--is worth the bother. Despite a major effort at making Output Measures for Public Libraries (Van House et al., 1987) easy to use, practical, and useful for library decision-making decision-making,
n the process of coming to a conclusion or making a judgment.
n a type of informal decision-making that combines clinical expertise, patient concerns, and evidence gathered from , some practitioners found that this, and similar assessment manuals, required too much time and effort to implement.
A major difficulty with outcomes assessment is that understanding outcomes, developing approaches to use outcomes as an assessment technique, training staff to be able to implement outcomes assessment, and then using the results for decision-making can be time-consuming time-con·sum·ing
Taking up much time.
taking up a great deal of time
Adj. 1. and difficult. Despite attempts for practical outcomes assessment manuals (e.g., Florida Florida, state, United States
Florida (flôr`ĭdə, flŏr`–), state in the extreme SE United States. A long, low peninsula between the Atlantic Ocean (E) and the Gulf of Mexico (W), Florida is bordered by Georgia and State Library, 2000) much work remains to be done to make outcomes assessment a practical, valid, and easy approach that will be embraced by the library community. Those developing such manuals need to work toward methodologies that are feasible, cost effective, and actionable Giving sufficient legal grounds for a lawsuit; giving rise to a Cause of Action.
An act, event, or occurrence is said to be actionable when there are legal grounds for basing a lawsuit on it. with relative ease. It is unclear at this time how well outcomes assessment--as a method--can be demystified and integrated easily into an ongoing library evaluation program.
Local Judgment Calls
Missing from a number of discussions about the importance of outcomes and outcomes assessment is the discussion of what are the "correct," "appropriate," or "right" outcomes for a particular library setting. Thus, the research question here is: What factors within the library and the local community affect the judgment of when a service is "good enough"? The issue becomes more complicated if the library has to accommodate its outcomes with those of a larger institution (university or city government). Moreover, once there is a selection of a particular outcome as appropriate for that setting, "how much" of that outcome does the library need to obtain to claim "success"? Similar issues occur in other types of assessment methods. The concern, however, is in method replacing value judgments on the part of the users of these methods.
As an example, the performance measure "correct answer fill rate," offers no guidance unless we know if a 50 percent correct answer fill rate is acceptable for a particular library or if a 75 percent correct answer fill rate is desired instead. Having a performance measure, or having an outcome, is not the same as agreeing on the quality or performance level that a library wishes to achieve on that outcome or performance level. It is the responsibility of the individual library or organization to develop a process that results in agreement that the quality standard for correct answer fill rate might be something like, "reference staff will answer quick fact and bibliographic bib·li·og·ra·phy
n. pl. bib·li·og·ra·phies
1. A list of the works of a specific author or publisher.
a. questions correctly 70 percent of the time."
The quality level or "standard" that a library or organization accepts as a benchmark A performance test of hardware and/or software. There are various programs that very accurately test the raw power of a single machine, the interaction in a single client/server system (one server/multiple clients) and the transactions per second in a transaction processing system. by which to judge itself depends on:
* Institutional and library goals and objectives (either developed internally or imposed externally);
* Stated priorities among those objectives (libraries cannot be all things to all people all the time);
* Resource allocations resource allocation Managed care The constellation of activities and decisions which form the basis for prioritizing health care needs among goals, objectives, and services;
* Staff knowledge and skills in providing information services; and
* Other situational factors at play at an individual setting.
The easy part of evaluation may be identifying and validating val·i·date
tr.v. val·i·dat·ed, val·i·dat·ing, val·i·dates
1. To declare or make legally valid.
2. To mark with an indication of official sanction.
3. the performance measures, outcome, or various statistics as indicators of information services. The hard part may be the development of a process within the organization to agree on the quality standards for those outcomes that are acceptable, or are the target, for that particular organization. In short, outcomes assessment has little utility without accompanying quality standards, and these require judgment calls on the part of the users of such methods.
Recognizing the Political Context of Outcomes Assessment
Evaluation, overall, has significant political overtones beginning with the determination of what to evaluate, how it will be evaluated, what outcomes are most important, what measures to use, how to interpret findings, and how best to report or present the findings. Noticeably no·tice·a·ble
1. Evident; observable: noticeable changes in temperature; a noticeable lack of friendliness.
2. Worthy of notice; significant. absent from a number of the discussions of outcomes assessment are considerations related to the political environment in which the library exists. A determination of "appropriate" outcomes by the larger institution can eliminate effectively the ability of the library to demonstrate how it supports those outcomes. In addition, these political considerations are compounded because of the network and technology infrastructure through which library services and resources are provided. For example, a public library that is dependent on a city's technology infrastructure for its network and resultant This article is about the resultant of polynomials. For the result of adding two or more vectors, see Parallelogram rule. For the technique in organ building, see Resultant (organ).
In mathematics, the resultant of two monic polynomials services has a more complicated political context for determining and measuring outcomes than a library that controls its own technology.
Dealing with the various political issues related to outcomes and outcomes assessment in a networked environment is new to most libraries. In addition, outcomes assessment in the highly charged technological infrastructure of many organizations requires the evaluator to have a range of skills and knowledge not needed in the nonnetworked environment. Research is needed to shed light on those serious issues.
As suggested in the key research questions posed earlier in this article, it is necessary to view research related to outcomes and outcomes assessment in the larger context of overall library and information services assessment. Outcomes assessment is but one method that has the potential to help library managers make better decisions regarding the provision of information services and resources. At present, however, there are many issues and problems with practical implementations of the approach. While there certainly is potential for developing outcomes assessment, much work--especially empirical research--is necessary.
As discussed previously, use of the terms outcomes, performance measures, quality standards, service quality, etc. continue to be disputed. As Kyrillidou (2002) concludes in a recent article, not only is there a need for more dynamic models to describe libraries and users, "we lack an adequate shared understanding of how outcomes are defined" (p. 45). The framework outlined in this article (see Figure 2) is one effort to help clarify the use of these terms and provide a means to understand their relationship to each other. Without better agreement among researchers about the use of these terms there is likely to be limited implementation of these various assessment techniques. Indeed, at a recent workshop conducted by the authors, a participant commented, after a review of the various assessment methods and approaches, that researchers and writers studying outcomes were like kids playing in a sandbox--and that clear, practical, usable USable is a special idea contest to transfer US American ideas into practice in Germany. USable is initiated by the German Körber-Stiftung (foundation Körber). It is doted with 150,000 Euro and awarded every two years. evaluation methods and approaches are not available to practitioners.
Also lacking from many library evaluation approaches is adequate recognition of how the networked environment affects services and resource provision. As shown earlier in Figure 1, the way online services are accessed raises a range of evaluation issues that require substantial attention. A key theme throughout this article is that outcomes assessments, as well as other types of library assessment, have yet to address factors and issues arising from the networked environment that seriously complicate com·pli·cate
tr. & intr.v. com·pli·cat·ed, com·pli·cat·ing, com·pli·cates
1. To make or become complex or perplexing.
2. To twist or become twisted together.
1. valid assessment approaches. Indeed, little of the outcomes assessment work to date considers the evaluation of network-based services or resources--and how those services/resources differ from traditional library services or resources. And yet, libraries are increasing the network-based services and resources that they provide.
There are a number of possible next steps for researchers working in the area of outcomes assessment:
* Develop a better understanding of outcomes and how outcomes "fit" into the range of evaluation models. This could be done by comparing the strengths and weaknesses of the models as shown in Table 2 and by reviewing the considerations given in this article that the models do not address currently.
* Specify and rethink re·think
tr. & intr.v. re·thought , re·think·ing, re·thinks
To reconsider (something) or to involve oneself in reconsideration.
re college and university accreditation documents to better determine what they mean by outcomes assessment and the process by which such assessment should be done. The review of such accreditation documents by Gratch-Lindauer (2002) clearly identifies confusion and lack of understanding by these accreditation agencies concerning outcomes assessment.
* Involve library practitioners in research related to outcomes to obtain their knowledge, views, and experiences in evaluation.
* Conduct empirical research on the validity of Table 3 in this article and determine the degree to which outcomes are similar or different in a traditional (print and face-to-face service provision) environment versus a networked environment.
* Conduct empirical research that determines the extent to which outcomes are generic and, thus, obtainable and comparable across a range of libraries and library types based on library situational and other factors.
* Sponsor a small meeting or symposium symposium
In ancient Greece, an aristocratic banquet at which men met to discuss philosophical and political issues and recite poetry. It began as a warrior feast. Rooms were designed specifically for the proceedings. in which key researchers and practitioners could debate and discuss a range of issues and concerns regarding outcomes assessment. Such a meeting could help focus attention on how best to attack outcomes assessment as an evaluation method.
These are only a few efforts possible to begin to address some of the research questions identified in this article. Until such steps are taken and a range of research activities initiated, outcomes assessment will continue to be an idea in search of both a theory and a practice.
There is much work yet to do in translating outcomes assessment and other evaluation approaches into practical, useful, and valid assessment techniques in the networked environment. The track record for the degree to which practitioners have embraced a program of ongoing evaluation (regardless of the method) is extremely low. Lakos (1999) identified the need for a "culture of assessment" within libraries years ago--and it is still lacking in large part. Needed now is more empirical research related to these evaluation approaches; evaluation methods that recognize and understand the role of the networked environment in the provision of services and resources; partnerships with libraries and related organizations to test, refine, and validate To prove something to be sound or logical. Also to certify conformance to a standard. Contrast with "verify," which means to prove something to be correct.
For example, data entry validity checking determines whether the data make sense (numbers fall within a range, numeric data practical and doable approaches; and a commitment to developing multiple evaluation methods that can work successfully together.
Table 1. Selected Definitions of Outcomes and Outcomes Assessment. * Citation Definition of Outcome Zweizig, D., Johnson, D., Robbins, J., "Outcomes--how things & Besant, J. (1994). TELL IT! changed for the Evaluation sourcebook and training community" (p. 104). manual. Madison, WI: University of Wisconsin, Madison, School of Library and Information Studies. United Way. (1996). Measuring program "Outcomes are benefits for outcomes: A practical approach. participants during or after Alexandria, VA: United Way. their involvement with a program. Outcomes may relate to knowledge, skills, attitudes, values, behavior, condition, or status" (p. xv). Himmel, E., & Wilson, W. J. (1998). "... a service response is what Planning for results: A public a library does for, or offers library transformation process: The to, the public in an effort to how-to manual. Chicago: American meet specific community needs. Library Association. Service responses represent the gathering and deployment of critical resources to produce a specific public benefit or result" (p. 51-52). Himmel, E., & Wilson, W.J. Planning "Evaluation is the process for results: A public library used to measure the transformation process: The performance of a guidebook, Chicago: American service against some Library Association. pre-determined criterion to see how well or poorly the service has been performed" (p. 35). Institute of Library and Museum Outcomes are "benefits or Services. (2000). Perspectives on changes for individuals outcome based evaluation for or populations during or libraries and museums. after participating in Washington, D.C.: Institute of program activities ..." Library and Museum Services. that are measured against predetermined criteria (p. 20). Free Library of Philadelphia. Outcomes are "the quality (2000, November 30). Free Library of things produced" as of Philadelphia: Performance related to the goals set management final report. forth early in the project's Philadelphia, PA: Price development (E, p. 1). Waterhouse Coopers. Association of College and Focus "on the achievement of Research Libraries. (2000). outcomes that have been Standards for college libraries identified as desirable in 2000 edition. Chicago: American the library's goals and Library Association, ACRL objectives. It identifies College Libraries Section performance measures, Standards Committee. such as proficiencies, Available at: http://www. that indicate how well the ala.org/acrl/guides/college.html library is doing what it has stated it wishes to do." Bertot, J. C., McClure, C. R., & "An outcome measure is Ryan, J. (2001). Statistics and explicitly tied to the performance measures for public libraries goals, objectives, library networked services. and planning process. A Chicago: American Library good outcome measure provides Association. data that tells a library manager if a specific library objective has been achieved" (p. 66). Hernon, P., & Dugan, R. E. (2002). "Outcomes assessment deals An action plan for outcomes assessment with academic institutions in your library. Chicago: providing evidence that they American Library Association. are meeting their educational mission. In the case of the library, outcomes focus on how library users changed as a result of their contact with the library and its resources, services, and programs" (p. 2). Fraser, B. T., & McClure, C. R. An outcome is a "clearly (2002, forthcoming). Toward a identified result or end framework for assessing product that occurs as a library and institutional consequence of individual outcomes. Portal: Libraries or combined activities from and the Academy, 2(4), 505-528. units of the institution. It is a preferred or desired state and ideally clarifies specific expectations of what should be products from the institution." * The authors wish to thank Lara Rudolph for her assistance in compiling the citations in this table. Table 2. Selected Outcome Models. * Citation Model Kyrillidou, M. (2002). From The Linear Model input and output measures to quality and outcome measure, or, from the user in the life of The Cyclic Model the library to the library in the life of the user. The Journal of Academic Librarianship, 28, pp. The Spiral Swirl 42-46. Bertot, J. C., & McClure, C. R. Outcomes and (2003). Outcomes assessment in Performance the networked environment: Measurement in Research questions, issues, Libraries considerations, and moving forward. Library Trends 51:4. Hernon, P. (2002). Outcomes Aggregate Factors are key but not the whole Approach story. The Journal of Academic Librarianship, 28, pp. 54-55. King et al. (2003). Library Derived Measures economic measures. Library Trends, 51 (3). Fraser, B. T., & McClure, Preliminary C. R. (2002). Toward a Framework of the framework for assessing Outcomes Assessment library and institutional Process outcomes. Portal: Libraries and the Academy, 2, pp. 505-528. Citation Summary Kyrillidou, M. (2002). From Based on the "assumption input and output measures to that inputs have a quality and outcome measure, direct relation to or, from the user in the life of outputs, which, in turn, the library to the library in the relates to quality life of the user. The Journal of and outcomes" (p. 44). Academic Librarianship, 28, pp. 42-46. Acknowledges that user transactions have "multiple dimensions of inputs, output, and quality and outcome elements from multiple interactive and reflective perspectives" (p. 44-45). "[T]ries to introduce the notion of motion depicting a more dynamic and flexible model, moving users and information resources into a spiral swirl up and down into the depths of knowledge, exploration, and experience" (p. 45). Bertot, J. C., & McClure, C.R. "Inputs and outputs form (2003). Outcomes assessment in the basis for service the networked environment: quality and outcomes Research questions, issues, assessment activities. considerations, and moving Outcomes assessment seeks forward. Library Trends 51:4. to determine the impact of the library's services/resources (again, outputs) on the library service and resource users. One cannot have outcomes measures without measures of outputs-- libraries need to know what investments (inputs) produce what services (outputs) in order to determine the perceived quality (quality assessment) and impacts (outcomes) of those services/resources" (p. X). Hernon, P. (2002). Outcomes Sees "library planning and are key but not the whole decision making as story. The Journal of Academic revolving around service Librarianship, 28, pp. 54-55. quality and its companion concept, satisfaction, and outcomes (or more precisely outcomes assessment). Inputs and outputs support the accomplishment of service, quality, satisfaction, and an assessment plan" (p. 54). King et al. (2003). Library "Measures are designed to economic measures. Library serve the perspectives of Trends, 51 (3). library staff management, library users, the fundors of the library and the higher order community served by the library" (X). "Outcomes are best determined by relating them to the purposes for which the information is obtained ..." (X). Fraser, B. T., & McClure, Model depicts "a basic C.R. (2002). Toward a process by which an framework for assessing academic research library library and institutional helps meet the goals of outcomes. Portal: Libraries particular departments and and the Academy, 2, functional units within a pp. 505-528. university, which in turn contribute to institutional goals, while acknowledging that libraries may also contribute more directly" (p. 519). * The authors wish to thank Kim A. Thompson for her assistance in compiling the citations in this table. Table 3. Outcomes and Outcome Levels. Outcome Outcome Level Time Dimension User Library Institutional System State National Anticipated Emergent Unanticipated
Association of Research Libraries. (2002). Measures for electronic resources (e-metrics). Unpublished announcement. Washington Washington, town, England
Washington, town (1991 pop. 48,856), Sunderland metropolitan district, NE England. Washington was designated one of the new towns in 1964 to alleviate overpopulation in the Tyneside-Wearside area. , D.C. Retrieved December December: see month. 12, 2002, from http:// www.arl.org/stats/announce/emetricspr.html.
Bertot, J. C. (2000). Libraries on the information highway: Issues and lessons learned. In P.D. Fletcher Fletcher may refer to one of the following: Ideas and companies
(2) A proposed high-speed communications system that was touted by the Clinton/Gore administration to enhance education in America in the 21st century. Its purpose was to help all citizens regardless of their income level. (pp. 288-300). Hershey Hershey, uninc. city (1990 pop. 11,860), Dauphin co., S central Pa.; founded 1903 as a planned community built for workers at the Hershey Corp., the chocolate manufacturer that remains its largest employer. , PA: Idea Group Publishing.
Bertot, J. C. (2001). Measuring service quality in the networked environment: Approaches and considerations. Library Trends, 49(4), 758-775.
Bertot, J. C., McClure, C. R., & Davis, D. M. (2002) Developing a national data collection model for public library networked statistics and performance measures: Interim report. Washington, D.C.: Institute of Museum and Library Services.
Bertot, J. C., McClure, C. R., & Ryan, J. (2001). Statistics and performance measures for public library networked services. Chicago Chicago, city, United States
Chicago (shĭkä`gō, shĭkô`gō), city (1990 pop. 2,783,726), seat of Cook co., NE Ill., on Lake Michigan; inc. 1837. : American Library Association American Library Association, founded 1876, organization whose purpose is to increase the usefulness of books through the improvement and extension of library services. .
Cook, C., & Heath, F. M. (2001). Users' perceptions of library service quality: A LibQUAL+ qualitative qualitative /qual·i·ta·tive/ (kwahl´i-ta?tiv) pertaining to quality. Cf. quantitative.
pertaining to observations of a categorical nature, e.g. breed, sex. study. Library Trends, 49(4), 538-584.
Cook, C., Heath, F. M., Thompson Thompson, city, Canada
Thompson, city (1991 pop. 14,977), central Man., Canada, on the Burntwood River. A mining town, it developed after large nickel deposits were discovered in the area in 1956. , B., & Thompson, R. L. (2001). The search for new measures: The ARL LibQUAL+ study--A preliminary report. Portal: Libraries and the Academy, 1(1), 103-112.
Dresang, E. T., & Gross, M. (2001). Evaluating children's resources and services in a networked environment. In C. R. McClure & J. C. Bertot, (Eds.), Evaluating networked information services: Techniques, policy, and issues (pp. 23-44). Medford Medford.
1 City (1990 pop. 57,407), Middlesex co., E Mass., a residential and industrial suburb of Boston, on the Mystic River; settled 1630, inc. as a city 1892. Wax, paper, clothing, and furniture are among its products. , NJ: Information Today.
Florida State Library. (2000). Workbook work·book
1. A booklet containing problems and exercises that a student may work directly on the pages.
2. A manual containing operating instructions, as for an appliance or machine.
3. : Outcome measurement of library programs. Tallahassee Tallahassee (tăləhăs`ē), city (1990 pop. 124,773), state capital and seat of Leon co., NW Fla.; inc. 1825. Tallahassee is a wholesale trade and distribution center for the surrounding lumber, livestock, and agricultural area. : Florida Department Florida is a department (departamento) of Uruguay. Population and Demographics
As of the census of 2004, there were 68,181 people and 21,938 households in the department. The average household size was 3.1. For every 100 females, there were 100.4 males. of State, Division of Library and Information Services.
Fraser, B., & McClure, C. R. (2002). Toward a framework for assessing library and institutional outcomes. Portal: Libraries and the Academy, 2(4), 505-528.
Gratch-Lindauer, B. (2002). Comparing the regional accrediation [sic Latin, In such manner; so; thus.
A misspelled or incorrect word in a quotation followed by "[sic]" indicates that the error appeared in the original source. ] standards: Outcomes assessment and other trends. Journal of Academic Librarianship, 28(1-2), 14-25.
Hernon, P. (2002). Outcomes are key but not the whole story. Journal of Academic Librarianship, 28(1-2), 54-55.
Hernon, P., & Dugan, R. E. (2002). An action plan for outcomes assessment in your library. Chicago: American Library Association.
Hernon, P., & Whitman, J. R. (2001). Delivering satisfaction and service quality: A customer-based approach for libraries. Chicago: American Library Association.
Institute of Museum and Library Services (IMLS). (2000). Perspectives on outcome based evaluation for libraries and museums. Washington, D.C.: Institute of Museum and Library Services. Retrieved December 4, 2002, from <http://www.imls.gov/pubs/pdf/pubobe.pdf>.
King, D. W., Boyce Boyce may refer to:
Montgomery, city (1990 pop. 187,106), state capital and seat of Montgomery co., E central Ala., near the head of navigation on the Alabama River just below the confluence of the Coosa and Tallapoosa rivers, and in the rich , C. H., & Tenopir. C. (2003). Library economic measures: Examples of the comparison of electronic & print journal collections and collection services. Library Trends, 51(3), 376-400.
Kyrillidou, M. (2002). From input and output measures to quality and outcome measures, or from the user in the life of the library to the library in the life of the user. Journal of Academic Librarianship, 28(1-2), 42-46.
Lakos, A. (1999). The missing ingredient
An ingredient is something that forms part of a mixture (in a general sense). : Culture of assessment in libraries. Performance Measurement and Metrics metrics Managed care A popular term for standards by which the quality of a product, service, or outcome of a particular form of Pt management is evaluated. See TQM. , 1(1), 3-7. Retrieved March 20, 2003, from http://www.aslib.com/pmm/1999/aug/opinion.pdf.
Lance, K. C., Steffen, N. O., Logan Logan, city (1990 pop. 32,762), seat of Cache co., N Utah, on the Logan River; inc. 1859. It is the center of an irrigated dairy and farm area, with huge cheese plants, other food-processing facilities, and diverse manufactures. , R., Rodney Rodney may refer to:
Aurora (ərôr`ə, ô–).
1 City (1990 pop. 222,103), Adams and Arapahoe counties, N central Colo., a growing suburb on the east side of Denver; inc. 1903. , CO: Bibliographical Center for Research The Bibliographical Center for Research, commonly known as BCR, is a "nonprofit, multistate library cooperative." It was founded in 1935 and now includes "more than 8,000 member libraries in 42 states, Canada and Guam. . Retrieved January January: see month. 6, 2003, from http://www.lrs.org/html/about/ CountingOnResults.htm.
McClure, C. R., & Bertot, J. C. (1998). Public library use in Pennsylvania Pennsylvania (pĕnsəlvā`nyə), one of the Middle Atlantic states of the United States. It is bordered by New Jersey, across the Delaware River (E), Delaware (SE), Maryland (S), West Virginia (SW), Ohio (W), and Lake Erie and New York : Identifying uses, benefits, and impacts. Harrisburg Harrisburg, city (1990 pop. 52,376), state capital and seat of Dauphin co., SE Pa., on the Susquehanna River; settled c.1710 by John Harris, who established a trading post and operated a ferry there; inc. 1791. , PA: Office of Commonwealth Libraries, Department of Education.
McClure, C. R., & Bertot, J. C. (Eds.). (2001). Evaluating networked information services: Techniques, policy, and issues. Medford, NJ: Information Today.
McClure, C. R., Fraser, B. T., Shim, W., & Bertot, J. C. (2002). Identifying and measuring library activities/services related to academic institutional outcomes. Washington, D.C.: Association of Research Libraries.
McClure, C. R., Lankes, R. D., Gross, M., & Choltco-Devlin, B. (2002). Statistics, measures, and quality standards for assessing digital reference library services: Guidelines and procedures. Syracuse Syracuse, city, Italy
Syracuse (sĭr`əkys, –kyz), Ital. Siracusa, city (1991 pop. , NY: Information Institute of Syracuse.
Shim, W.J., & McClure, C. R. (2002). Improving database vendors' usage statistics reporting through collaboration Working together on a project. See collaborative software. between libraries and vendors. College & Research Libraries, 63(6), 1-16.
Shim, W.J., McClure, C. R., Fraser, B. T., & Bertot, J. C. (2002). Data collection manual for academic and research library network statistics and performance measures. Washington, D.C.: Association of Research Libraries.
Smith, K. R. (2000). New roles and responsibilities for the university library: Advancing student learning through outcomes assessment. Washington, D.C.: Association of Research Libraries. Retrieved November November: see month. 8, 2002, from http://www.arl.org/stats/newmeas/outcomes/HEOSmith.html.
Smith, M., & Rowland, G. (1997, May-June). To boldly go
To Boldly Go (commonly known as TBG : Searching for output measures for electronic services, Public Libraries, 168-172.
United Way of America United Way of America: see community chest. . (1996). Measuring program outcomes: A practical approach. Alexandria Alexandria, city, Egypt
Alexandria, Arabic Al Iskandariyah, city (1996 pop. 3,328,196), N Egypt, on the Mediterranean Sea. It is at the western extremity of the Nile River delta, situated on a narrow isthmus between the sea and Lake Mareotis (Maryut). , VA: United Way of America.
Van House, N., Lynch, M.J., McClure, C. R., Zweizig, D. L., & Rodger Rodger is a surname, and may refer to:
pertaining to data that have been submitted to standardization procedures.
standardized morbidity rate
see morbidity rate.
standardized mortality rate
see mortality rate. procedures. Chicago: American Library Association.
John Carlo Bertot, Associate Professor and Associate Director, and Charles Charles, archduke of Austria
Charles, 1771–1847, archduke of Austria; brother of Holy Roman Emperor Francis II. Despite his epilepsy, he was the ablest Austrian commander in the French Revolutionary and Napoleonic wars; however, he was handicapped by R. McClure, Francis Francis, French prince, duke of Alençon and Anjou
Francis, 1554–84, French prince, duke of Alençon and Anjou; youngest son of King Henry II of France and Catherine de' Medici. Eppes Eppes may refer to several American people:
Louis, 1682–1712, titular duke of Burgundy; grandson of King Louis XIV of France. He became heir to the throne on the death (1711) of his father, Louis the Great Dauphin. Shores Building, Florida State University Florida State University, at Tallahassee; coeducational; chartered 1851, opened 1857. Present name was adopted in 1947. Special research facilities include those in nuclear science and oceanography. , Tallahassee, FL 32306-2100
JOHN CARLO BERTOT is Associate Professor and Associate Director of the Information Use Management and Policy Institute in the School of Information Studies at Florida State University. He teaches courses in library technology planning, technology applications, and information and telecommunications Communicating information, including data, text, pictures, voice and video over long distance. See communications. policy. Bertot has published extensively in the areas of library management, planning, and evaluation, with a particular emphasis on library Internet use and involvement. Most recently, he coauthored Statistics and Performance Measures for Public Library Networked Services through the American Library Association (2001), and coedited Evaluating Networked Information Services, an Information Today publication (2002). At present, Bertot is co-principal investigator for a national U.S. study sponsored by the Institute of Museum and Libraries to develop a national model for the collection and reporting of public library network statistics and performance measures as well as for the development of training modules to assist libraries in collecting, using, and reporting network-services data. Bertot also serves as a U.S. delegate A person who is appointed, authorized, delegated, or commissioned to act in the place of another. Transfer of authority from one to another. A person to whom affairs are committed by another.
A person elected or appointed to be a member of a representative assembly. on the International Standards Organization's (ISO (1) See ISO speed.
(2) (International Organization for Standardization, Geneva, Switzerland, www.iso.ch) An organization that sets international standards, founded in 1946. The U.S. member body is ANSI. ) Library Statistics committee, and is the convener con·vene
v. con·vened, con·ven·ing, con·venes
To come together usually for an official or public purpose; assemble formally.
1. of the ISO Library Performance Indicator committee. He is also a member of the National Information Standards Organization's (NISO (National Information Standards Organization, Baltimore, MD, www.niso.org) A non-profit organization founded in 1939 that deals with bibliographic and related information standards. ) planning committee planning committee n (in local government) → comité m de planificación for the revision of the Z39.7 Library Statistics standard.
CHARLES R. MCCLURE, who obtained his Ph.D. from Rutgers University Rutgers University, main campus at New Brunswick, N.J.; land-grant and state supported; coeducational except for Douglass College; chartered 1766 as Queen's College, opened 1771. Campuses and Facilities
Rutgers maintains three campuses. , is the Francis Eppes Professor of Information Studies at the School of Information Studies, Florida State University. He is also the Director of the Information Use Management and Policy Institute at Florida State University. McClure was the co-principal investigator on the Association of Research Libraries' funded E-metrics project, 2000-01; he was the co-principal investigator on a study funded by the U.S. Institute for Museum and Library Services to develop and test measures for networked services, 2000-02; and he recently completed a study funded by OCLC OCLC - Online Computer Library Center and a consortium of other library organizations that resulted in a manual to assess digital reference services. He is the coauthor co·au·thor or co-au·thor
A collaborating or joint author.
tr.v. co·au·thored, co·au·thor·ing, co·au·thors
To be a collaborating or joint author of: "He and a colleague . . . of Statistics and Performance Measures for Public Library Networked Services (American Library Association, 2001); Evaluating Networked Information Services (Information Today, 2002); and Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services (Syracuse University Press Syracuse University Press, founded in 1943, is a university press that is part of Syracuse University. External link