Printer Friendly

Outcomes assessment in the networked environment: research questions, issues, considerations, and moving forward.


THIS ARTICLE IDENTIFIES A NUMBER OF RESEARCH TOPICS related broadly to outcomes assessment in a networked environment and discusses issues affecting these research topics. It also proposes a framework to relate traditional evaluation components and terminology to the networked environment and identifies a number of factors in the networked environment that affect outcomes and other assessment methods. The article suggests that outcomes assessment has the potential to complement other assessment techniques to better assist libraries and related information organizations enhance their decisions in the provision of information services and resources. Given the increased rate at which libraries are using the networked environment to provide services and resources, however, much work remains before most libraries can implement outcomes assessment efforts successfully.


Organizations of many types in various operating environments have used a variety of performance assessment activities for some time. Libraries are no different, having engaged in the use of output, service quality, performance indicator, balanced scorecard, and a number of other performance measurement techniques. At the core of these measurement activities are a number of basic questions:

* What resources are required to support the services that a library provides?

* What services and/or resources is a library able to provide with its investments in library infrastructure (broadly defined as personnel, technology, collections, facilities, etc.)?

* Are the library's customers receiving value out of the community's investment in library services?

* What is the real and/or perceived quality of library services?

* What are the impacts of library services and resources on the community that the library serves?

* What level of effort is required by library staff to implement measurement activities?

* In what ways does the library contribute to the overall mission, goals, and objectives of the community--or institutions within that community--the library serves?

The role and usefulness of outcome measures and outcomes assessment in answering these and related questions, although potentially rich, requires significant additional review, research, field testing, and development.

The purpose of this article is to:

* Identify selected research questions regarding outcomes assessment;

* Review a number of issues affecting outcomes assessment in general and outcomes assessment in a networked environment in particular; and

* Propose a framework to relate and better understand traditional evaluation components and terminology.

The article suggests that outcomes assessment has the potential to evolve to a point at which it will complement other assessment techniques that assist libraries and related information organizations make better decisions in the provision of information services and resources. There are, however, a number of research, methodological, and other issues that require consideration prior to reaching this potential. Given the library community's increased reliance on providing services and resources via the networked environment, much work lies ahead before libraries can engage successfully in outcomes assessment activities.


Outcomes assessment is a relatively new activity to libraries, and as such, this is a very broad area for research. Indeed, as with any new area of scholarly endeavor, there are many more questions than answers. Some key research questions are:

* What are the prevailing models/approaches for library services/resources assessment?

* Are these models distinct or interrelated?

* To what degree do these models/approaches consider networked services/resources?

* How are "outcomes" considered in these various models?

* What do these models/approaches tell researchers and practitioners about use, uses, quality, impact, etc. of library services/resources?

* What outcomes assessment methods will be most useful and successful given the needs of a library, the resources available to the library for assessment, the assessment activity objectives, staff skills available, and other situational factors?

* What are the key variables to consider in the development of such models?

This article provides a foundation upon which to begin addressing these questions. The authors do not claim, however, that these are the only research questions that require attention regarding library outcomes assessment activities. These questions are, though, essential to further the development of outcomes assessment approaches that provide libraries with techniques that identify the impacts and benefits of library services/resources on the communities that they serve. The remainder of the article presents the issues and considerations regarding these research questions as a means of moving the research agenda forward.

The Current Context and the Need for Answers

The environment in which libraries find themselves at present creates a situation in which answers to the above posed research questions are essential. Libraries of all types, but public and academic libraries in particular, find that they are considered increasingly as part of the larger organizational structure in which they reside--university departments or local government agencies. As such, library funding and continued well-being is no longer distinct from that of any other campus or local agency. Because of this situation, libraries are being asked to:

* Articulate the importance of and need for their services and resources;

* Identify the use and uses of their services and resources; and

* Establish the value, impacts, and benefits that the community receives from the library services and resources.

Given the above circumstances, it behooves the library practitioner and research communities to have assessment tools and approaches that enable libraries to articulate their contribution to the well-being of the communities that they serve.

In addition to this management context, there is the evolving technology context in which libraries operate. To state the obvious, library services and resources rely increasingly on:

* Technologies that continue to change and evolve at an ever-intensifying pace. On the one hand, these technology changes enable new services and resources that allow libraries to better meet the service and resource needs of their customers. On the other hand, this continual cycle of adoption, change, and new and/or enhanced services and resources creates a number of challenges--including assessment challenges--for libraries.

* Leased and network-based resources and situations that remain outside the direct control of libraries (see Figure 1). For example, libraries continue to grow their subscriptions to online databases and resources (e.g., e-books). Moreover, customer access to those services and resources can occur through a number of nonlibrary venues--each of which has implications for what libraries and vendors can collect and report regarding those user-initiated activities. As such, libraries are in a situation in which they do not control the services themselves (libraries are merely subscribers to resources such as EbscoHost, ScienceDirect, and NetLibrary), do not control the path through which customers gain access to the service/resource (remote access is beyond the scope of the library), and do not control the customer service/resource use and interaction data. All of this has a direct impact on the assessment activities in which a library can engage, the findings derived from such activities, and the types of questions that libraries can ask users regarding their use and/or assessment of the services/resources accessed (Shim & McClure, 2002).


Added to this mix is the impression that some outside the library community regard the information content as available without charge to Internet users. As libraries struggle to identify, define, and articulate the impacts derived by customers through the use of library services and resources, libraries must educate the communities that they serve regarding the issues associated with networked-based information content.

There are a number of evaluation methodologies that exist--for example, return on investment, quality assessment, outcomes assessment, outputs, best practices--to assist libraries in identifying and communicating their service/resource impacts on their communities. There are also a number of situational factors that affect a library's or a researcher's ability to identify, study, and present results of that work. This article describes a selected number of those methodologies, explores the relationship between them, and presents issues associated with methods and what they can tell us about library services and resources.


A first step in addressing the research questions posed earlier is defining and understanding an "outcome." There is no single concise definition of what an "outcome" means in the context of library service. During the past decade, a number of writers and researchers have proposed definitions that Table 1 summarizes. A review of various definitions, however, does yield a number of common elements. In general, outcomes:

* Include the notion of an impact, benefit, difference, or change in a user, group, or institution based on the use of or involvement with a library service or resource;

* Are predetermined based on a service/resource planning process in which the library engages to produce desired service/resource outcomes through the setting of service/resource goals and objectives; and

* Involve measuring and demonstrating the extent to which library services/resources meet the anticipated outcomes determined by the library or imposed by the community the library serves (e.g., academic institution, county, city).

These definitions broadly, therefore, consider outcomes assessment to be a proactive endeavor on the part of the library in which there is an a priori determination of the library service/resource preferred outcomes.

In general, these definitions assume that there is a larger context in which libraries reside that provides the basis for the library service/resource outcomes. In the case of academic libraries, for example, the desired outcomes might take the form of accreditation standards set forth by accreditation bodies; accountability measures imposed by university administrators/ boards; or state government-imposed higher education outcomes. Finally, these definitions assume that libraries are able to measure their service/resource outcomes with reliable and valid instruments and demonstrate subsequently that library service/resource outcomes contribute to the mission, goals, and objectives of the community that the library serves.

Over time, library researchers, managers, and others have developed a number of models to describe and relate inputs, outputs, service quality, outcomes, impacts, etc. Kyrillidou (2002) summarizes some of these approaches. There is no lack of proposed models to describe these evaluation components (e.g., Hernon, 2002, p. 55; Cook & Heath, 2001, p. 580; Dresang & Gross, 2001, p. 28). While some might suggest that the plethora of views, models, and definitions describe a healthy intellectual development in library/information services and resources evaluation, others--namely many practitioners--may not agree. Part of the issue is that each of these views, models, and evaluation approaches is presented individually without any review or consideration of the relationship between the key aspects of these models or perspectives.

Outcomes assessment, although not new to other organizations and sectors, is relatively new to libraries and focuses on determining the impact of a library's services and/or resources on its customers. In the broadest sense, outcomes assessment focuses on the extent to which a library's services and/or resources made a difference in the life of the library's individual, group, or institutional users (see Table 2). Writers and researchers, however, do not agree on what it means to measure outcomes.

Kyrillidou (2002) presents three models that depict the possible relationship between inputs, outputs, outcomes, and service quality. These models include a linear model, cyclic model, and spiral swirl (p. 44). Without empirical testing, Kyrillidou finds problems with all the models presented save the spiral swirl as it apparently can depict motion and a more flexible and dynamic process that intertwines inputs, outputs, outcomes, and service quality. Ultimately, Kyrillidou concludes that service quality is an outcome of library services and resources that exist through a library resource input and output process.

Hernon (2002) reaches a different conclusion, stating that service quality and satisfaction and outcomes assessment "truly stand out as central assessment concepts for librarianship. Inputs and outputs are important to institutions and accrediting bodies as measure of efficiency and crude measure of effectiveness" (p. 55). In Hernon's view, therefore, there is little to no relationship between inputs, outputs, service quality, and outcomes assessment--and libraries should focus on service quality and outcomes assessment techniques. King et al. (2002) take a different approach by linking types of measurement perspectives (library, user, organization, etc.) with specific types of measures--one of which is outcomes. The approaches shown in Table 2 are illustrative of recent writings, though historically many writers have offered models and approaches related to inputs, measures, service quality, and outcomes.

Table 2, based on research by the authors and others, summarizes the view of the outcomes assessment environment of public and academic libraries from selected perspectives. Each of the described approaches has strengths and weaknesses; each makes implicit assumptions about outcomes and the relationships between outcomes and other evaluation factors; and most do not address a range of factors and considerations discussed later in this article. In short, it is not possible to address meaningfully the research questions posed initially in this article with such different views of outcomes and the relationship of outcomes to other evaluation approaches.

The authors of this article present a view summarized in Figure 2. This model, based on a number of large-scale field-tested and empirical research efforts, posits that inputs and outputs form the basis for service quality and outcomes assessment activities (Bertot, McClure, & Davis, 2002; Shim et al., 2002; Bertot, McClure, & Ryan, 2001). In this view, inputs are the resources that libraries invest (e.g., money, staff, workstations, online commercial databases) in order to produce outputs (e.g., number of users of the workstations, number of database content downloads, circulation of material). Quality assessment involves determining the degree to which users find the library services/resources (outputs) to be satisfactory. Outcomes assessment, however, seeks to determine the impact of the library's services/resources (again, outputs) on the library service and resource users. In other words, one cannot have outcomes measures without measures of outputs--libraries need to know what investments (inputs) produce what services (outputs) in order to determine the perceived quality (quality assessment) and impacts (outcomes) of those services/resources.


An Outcomes Framework--More than One Type of Outcome

Much of the outcomes literature presents outcomes as monolithic--that is, there is a set of generic outcomes that fit various organizational settings (IMLS, 2000). However, it is important to recognize that there are many types of outcomes, such as research and learning outcomes (Hernon & Dugan, 2002) as well as institutional outcomes (Fraser & McClure, 2002). Developing a framework for outcomes assessment requires a complex analysis that encompasses the operating environment of the library, the impact of situational factors on library services and resources outcomes, and the reality that it is not always possible for libraries to anticipate and/or predict the outcomes of their services/resources on users.

Different types of outcomes have been suggested by Lance et al. (2002), Hernon and Dugan (2002), Bertot (2001), and Fraser and McClure (2002)--to name a few. As a result, developing a framework for outcomes assessment requires a complex analysis that considers a number of outcome types:

* Economic. Economic outcomes would include the impact of library services and resources on the ability of library users to prosper financially, seek employment successfully, or develop and sustain a small business in the community;

* Learning. Learning outcomes would include the impact of library services and resources on the ability of library users to engage in lifelong learning, interact with and engage a number of information resources, develop information literacy skills, develop technology skills, become literate, or develop an analytic ability to assess the validity and reliability of information sources;

* Research. Such outcomes might include the impacts of library services and resources on the research process of faculty and students in a university--e.g., assistance in proposal writing, grant receipt, and publication;

* Information Exchange. Information exchange outcomes would include the impact of library services and resources on the ability of library users to interact with government, exchange information with distant family, or receive information regarding countries of interest (e.g., foreign newspapers and other sources);

* Cultural. Cultural outcomes would include the impact of library services and resources on the ability of library users to develop an appreciation for fine arts, history, music, diversity, or other societal aspects; and

* Community. Some library outcomes affect the local community, be that an academic setting, a city or town, or a virtual community. Such outcomes could affect the overall quality of life for members of the community, attitudes of community members toward services, or even the political landscape of the community.

These categories are not mutually exclusive but rather are intended to be illustrative of types of outcomes. It is possible to identify other types of outcomes, as well as to expand the nature of the outcomes categories presented above. The key point is, however, that outcomes can be many and varied--and produced by the same service or resource (e.g., public access Internet workstations, online databases).

As identified in Table 3, there are two additional aspects of outcomes that researchers and practitioners need to consider: the level of outcome applicability and the time dimension of the outcome itself (discussed below). It is important to note that outcome types may apply at different levels (illustrated by the columns in Table 3). For example, a learning outcome may apply to the:

* Individual user of the library's services/resources by gaining the ability to read;

* Library by having a now-literate customer consume other library services/resources and derive additional impacts/benefits;

* Community by now having a literate member who can seek and expand his/her employability; and

* State and nation through a more economically solvent individual who contributes more to the economy through higher wages.

As such, outcomes can have multiple impacts for the individual user, library, library institution (e.g., university), system (i.e., state university system, public library system as a whole), state, and nation.

According to some authors, outcomes assessment implies that outcomes are predefined through a planning process in which libraries engage (Hernon & Dugan, 2002; Fraser & McClure, 2002). In this view, outcomes are established a priori through service and resource goals and objectives developed through a planning process; there is some type of pretest in which libraries ascertain the extent to which their services/resources meet these goals and objectives according to users; and then, after user exposure to the new and/or modified library services/resources, the library measures the impact (outcome) of the services/resources through a posttest. Again, this is an oversimplification of the outcomes environment. Indeed, it is possible to have outcomes that are on different time dimensions as the rows in Table 3 demonstrate. Outcomes can be:

* Anticipated. These are the outcomes for which the library plans and by which the library intends to measure its success/failure in goals and objectives attainment. The library expects to achieve certain outcomes through its services/resources and then seeks to ascertain the extent to which its services/resources achieved the anticipated outcomes. It is important to note that anticipated outcomes can be generated by the library or imposed externally by funding agencies (e.g., the Institute of Museum and Library Services) or governing authorities (e.g., university boards and provosts, accreditation bodies, city council, etc.). Most outcomes work to date focuses on this type of outcome assessment model.

* Emergent. These are outcomes that emerge through the service/resource planning and implementation process. Such outcomes are not the immediate focus of the service/resource goals and objectives--either library or externally imposed. However, as the library develops its service/ resource plans, these additional outcomes become apparent and are incorporated into the assessment process.

* Unanticipated. Once a service/resource is in operation, there are those outcomes that derive from actual service/resource use or interaction and can be ones that neither the library nor others predicted--nor planned to assess. For example, in the early years of public library Internet connectivity, many public librarians anticipated that the primary outcome of Internet connectivity in their libraries would be for information seeking, and quickly discovered that the primary outcome was communications--the ability of users to stay in touch with family and countries (McClure & Bertot, 1998). This outcome is still prevalent in public libraries, as demonstrated recently by Lance et al. (2002). By the same token, universities enhanced their Internet connectivity in their goal to become "wired campuses." Of the many anticipated outcomes
 associated with robust connectivity, one unintended outcome was the
 substantial use of university resources by students to access online
 music services (some later deemed illegal). A number of universities
 now block access to such services from their campuses to preserve
 their connectivity resources, as well as to limit the universities
 from liability due to various copyright infringements.

It is clear that outcomes are not always anticipated either by librarians or others with expectations for library service/resource use. By focusing on the anticipated outcomes, librarians (and others) ignore significant emergent and unanticipated outcomes--both positive and negative. Any outcomes assessment strategy, therefore, needs to incorporate the ability to discover and articulate outcomes that were not the intended focus of the services/resources--something not likely in a pre- or posttest quasi-experimental methodology.

Outcomes Assessment Methodology Issues

Embedded within the discussion of the types of outcomes identified above is the notion of gap identification. In an ideal situation, a library anticipates a set of outcomes from its services/resources, and the library's user community confirms those anticipated outcomes. It is more likely the case, however, that outcomes reside within a zone of tolerance in which library intended outcomes and user assessment of those outcomes are within an acceptable range--or minimal gap. There may be multiple gaps or zones of tolerance--the library, the university, the user--all of these constituencies comprise the outcomes space within which the library's services/resources reside. There is a need to explore further the notion of gap analysis in outcomes assessment activities such as is being done with service quality research activities (Cook et al., 2001).

Another critical factor in outcomes assessment is the ability to isolate the actual impact of library services/resources on the user community. As an example, we use one of Smith's (2000) institutional learning outcomes for university libraries, Develop Attitudes of Openness, Flexibility, Curiosity, Creativity, and an Appreciation of the Value of a Broad Perspective. Let us create a formula based on general linear model (GLM) analysis techniques for which one might use analysis of variance (ANOVA) or multiple regression to analyze:

Dependent Variable = Students develop attitudes of openness, flexibility, curiosity, creativity, and an appreciation of the value of a broad perspective

Independent Variable (s) = Particular library services/resources + courses + student extracurricular activities + ... +?

In quantitative terms, researchers as well as librarians and university officials will want to know what library services/resources contribute, in what way, and how much, to the student body's development of attitudes of openness, flexibility, curiosity, creativity, and an appreciation of the value of a broad perspective. In other words, the assessors of program outcomes will want to know which library services/resources contribute to the attainment of the outcome, as well as by how much, and in what way(s).

Needless to say, it is near impossible to quantify, much less isolate, a university library's contribution toward the outcome identified above. Simply put, student "attitudes" are formed by any number of campus activities in which students might engage--from coursework to mentoring by faculty to clubs to which the student may belong, etc. Library services/resources do not exist in a vacuum and measurement techniques will need to take these external and likely contributing factors into account. This is particularly the case when outcomes are presented as an institutional (i.e., university) objective but measurement occurs locally (i.e., at the library). To ignore the other university activities that likely contribute to the outcome would yield an incomplete picture of library contributions to outcome attainment at best and distort the library's contribution to outcomes attainment at worst.


Libraries now reside in a complex service environment--one that requires that they provide traditional services and resources such as a physical space, print material, and face-to-face reference, as well as network-based services such as Web-based collections, online databases, and virtual reference (Bertot, 2000; McClure, Fraser et al., 2002). Libraries are, therefore, providing more services and resources through multiple delivery methods. Moreover, while these traditional and network-based services are perhaps related and similar in function, they differ vastly in a number of significant ways such as the infrastructure required to deliver the services, the ways in which users or user communities access the services, the skills and knowledge required by uses or user communities to use the services, the reach and range of the services, the ways in which libraries manage such services, and the skills required within the library to deliver and access such services.

The networked environment is an intricate series of networks that interconnect in such a way that users never completely know the route they took to retrieve the information or resource that they sought. As Smith and Rowland (1997, p. 170) note, local users can access local resources, remote users can access local resources, and local users can access remote resources. Such a multifaceted environment--one in which the library is the resource or the library is the resource gateway for local or remote users--creates a complex environment for measurement. Data collection and research activities, therefore, need to reflect this complexity and multidimensionality of electronic networks. They also need to recognize the need for collecting a range of data for use at the organizational (library), institutional (university, county government), state, or national levels.

The implications for outcomes assessment due to this environment are numerous. They include, but are not limited to:

* Many library network-based services are not truly the services of that library. For examples, libraries license the use of such vendor-based services as EbscoHost and ScienceDirect. As such, they do not own, but rather lease, such "collections." Therefore, any measurement activities that assess the outcomes of library services need to take into account that libraries do not have direct control over such services. Thus, how best to "count" basic networked service interactions is complex (Shim & McClure, 2002).

* Users can and do access network-based services from numerous locations using a wide range of technology and telecommunications services. Each of these variations in access can and does affect the experience of the user--and these variations are beyond the purview of the library. Indeed, some library users do not access the library physically.

* The type of interaction that users can have with a library's network-based services and resources versus a library's physical services and resources can be quite different. Virtual users do not "browse the stacks" as do physical users. Moreover, not all of a library's collection is available in digital format; thus, virtual users are often limited to certain types of resources and services. In some cases, a service or resource is only available in electronic format. Another service that libraries provide increasingly is user training (e.g., use of technology, use of databases, other). Online tutorials differ greatly from in-person training in a library lab setting.

These issues suggest that research activities cannot regard all users as the same, nor can they treat all library services and/or service delivery modes as the same.


As identified earlier in this article, the discussion of outcomes and outcomes measurement in libraries and related information organizations is in its infancy. There are more questions about outcomes and outcomes assessment than there are adequate answers at this point. Indeed, it may be that at some point in the future, researchers will determine that outcomes and outcomes assessment have little utility in library organizational and services evaluation because of problems of implementations--especially in a networked environment. The research questions posed in the beginning of this article only scratch the surface of those that require attention. As the research and library communities move forward in studies related to outcomes and outcomes assessment, there is a need to make explicit some assumptions, considerations, and concerns.

Need for Empirical Research

At present, there is no lack of opinions and views related to outcomes and outcomes assessment. But, there is a lack of empirical research related to outcomes, outcomes assessment, and conceptual frameworks that would help practitioners to understand better the basis for, application of, and use of outcomes assessment. One empirical effort was the Association of Research Libraries' sponsored E-metrics study (Association of Research Libraries, 2002). One aspect of the E-Metrics project explored the role of the academic library in its contribution to achieving institutional outcomes. Findings from the E-Metrics study suggest (Fraser and McClure, 2002):

* The inputs-outputs model for library assessment (which produced the statistics and measures in the report Measures and Statistics for Research Library Networked Services: Procedures'and Issues) may not be linked easily to demonstrating the library's role in accomplishing institutional outcomes.

* Many of the library's activities, resources, and services are combined with other institutional activities, resources, and services in such a way that parsing out only the library's contribution to institutional outcomes is extremely difficult.

* The process by which libraries are involved in the identification of and agreement to these institutional outcomes is not clear.

* There is widespread confusion as to what an "institutional outcome" is and how such outcomes "fit" into traditional assessment procedures.

* Increasingly, academic accreditation agencies are considering the use and appropriateness of "institutional outcomes" as a means to assess the degree to which the organization determines what end products should result from organizational activities.

* The issue of nonexistent or inconsistent incomparable usage statistics provided by external information content providers (vendors) is a major stumbling block for libraries to gauge rapidly increasing use of electronic resources by research library users, thus making it difficult to use the data as sources for establishing library outcomes.

To a large degree, the E-metrics study found that academic libraries are not well prepared to demonstrate the extent to which they contribute to the organization's accomplishment of institutional outcomes. Moreover, libraries are looking at outcomes at the library level when, in fact, they reside within an institutional context. Nonetheless, there appears to be considerable interest in identifying and measuring the degree to which the academic library does contribute to accomplishing such institutional outcomes.

One of the products from the E-metrics study was a proposal for additional research in the area of outcomes, "Identifying and Measuring Library Activities/Services Related to Academic Institutional Outcomes" (McClure et al., 2002). The proposal outlined a conceptual framework for understanding outcomes assessment that has yet to be tested. In fact, there are few efforts to propose conceptual frameworks (such as that offered in Figure 2 earlier in this article) to help researchers lodge such research in the broader frame of evaluation theory. Thus, empirical research projects are necessary to understand better and describe outcomes and outcomes assessment--regardless of the type of outcomes being investigated.

Need for Multimethod Approaches

From the issues identified and discussed in this article, it is clear that research endeavors cannot treat all users as the same, all delivery of library services in the networked environment as the same, nor should they lump generically library services for assessment purposes. The differences between online and physical services/resources are real and not comparable (Bertot, McClure, & Davis, 2002; Shim et al., 2002; Bertot, 2001). Research efforts that do not separate network-based library services and resources from physical services/resources--such as work by Hernon and Dugan (2002) and Hernon and Whitman (2001)--miss substantive differences and, ultimately, the ability to determine library service and resource quality and outcomes.

While there is a need to focus much research on outcomes, such research should not occur at the exclusion of promising work in other areas of assessment. For example, there is a range of other efforts currently underway to assist the library community in the assessment of services and resources. These other efforts focus on:

* Service discrepancies among different stakeholder groups (e.g., LibQUAL+);

* Inputs analysis;

* Outputs analysis;

* Cost/benefit analysis;

* Performance measures (that can combine inputs and outputs); and

* Quality standards.

The issue is not which of these, or others, is the best approach for libraries. The issue is better stated as: Given the needs of the library, the resources available to the library for assessment, the assessment activity objectives, and the staff skills available--what methods for assessment will be most useful and successful?

Oftentimes, a combination of methods is best as each method has its own strengths and weaknesses. For example the recently published manual Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Procedures (McClure et al., 2002) offers some thirty-five performance measures related to the provision of digital reference services as well as six suggested areas for quality standards. These measures and quality standards are a first comprehensive effort for assessing digital reference services. Inclusion of cost-benefit analysis, service quality approaches, and outcomes assessment could also be incorporated with the approach described in the manual. As methods are integrated and expanded, however, the cost and level of effort required to conduct the assessment increases.

Multimethod approaches are also likely to be most useful given the range of situational factors that affect individual organizations and libraries. The type of assessment required for a small rural public library that is a member of a large regional cooperative with few staff and resources is likely to vary considerably from a large academic library that purchases an extensive collection of electronic resources, has talented technology staff, and provides customized networked services to its academic community.

In the experience of these authors, one evaluation method does not fit all types of libraries. The promotion of one particular method by some at the expense of all other methods does not reflect the complexity of situational factors as they relate to assessment in a library context. Nor might a single approach provide reliable and valid research results in all library organizational settings--there is a need for flexibility in methods and execution for libraries to engage successfully in evaluation activities.

Impact of the Networked Environment

As discussed earlier in this article, assessment of library services and resources--regardless of method--needs increasingly to consider the role and impact of electronic services and resources. Assessment of electronic services and resources, especially in a networked environment, involves a very different set of circumstances than those in the print and physical environment (McClure & Bertot, 2001). Assessment in a networked environment raises the following unique issues:

* The technology for the delivery of these services and resources is constantly changing, which affects what services/resources a library can deliver and how such delivery might occur. This also affects the methods, approaches, and software that enable the "tracking" of such services/resources;

* There is a need to have comparative (and valid) measures for traditional versus electronic-based services and resources, e.g., is a physical visit to the library (turnstile count) comparable to a visit to the library's Web site?

* Oftentimes, libraries contract (license) for networked services or do not control their own technology infrastructure (e.g., servers), thus they cannot easily obtain transactional data describing services and resource provision;

* The increased reliance on consortia and other group arrangements for the purchase and delivery of electronic services and resources blurs actual costs for these services and resources as well as how these services and resources are being used;

* Related to the above, leased services/resources are not truly the services/resources of the library. Rather, the library acts as a gateway to such services/resources. Outcomes assessment activities need to consider the most appropriate strategies for evaluating the outcomes of services/resources that are technically not the library's; and

* Librarian skills necessary to conduct outcomes and other types of assessment in the networked environment are significant and require constant enhancement given the changing nature of that environment.

Ease of Use

One major concern regarding the development of outcomes assessment methods is the ability of libraries to implement the recommended approaches, data collection activities, interpret the results, and use the findings to inform library resource/service planning efforts as well as external library stakeholders. Thus, it is imperative that researchers engaged in the development of outcomes assessment tools consider the degree to which the methods are practical and feasible for day-to-day use in libraries and other information organizations. As this article indicates, however, the ability of researchers to meet this ease-of-use burden remains unclear as outcomes assessment requires complex indicators and methods.

To some degree, outcomes assessment is a theory in search of a practice. From the earliest days of the development of various output measurement manuals in the early 1980s, many practitioners remain unconvinced that the amount of time and resources necessary for conducting assessments--of any type--is worth the bother. Despite a major effort at making Output Measures for Public Libraries (Van House et al., 1987) easy to use, practical, and useful for library decision-making, some practitioners found that this, and similar assessment manuals, required too much time and effort to implement.

A major difficulty with outcomes assessment is that understanding outcomes, developing approaches to use outcomes as an assessment technique, training staff to be able to implement outcomes assessment, and then using the results for decision-making can be time-consuming and difficult. Despite attempts for practical outcomes assessment manuals (e.g., Florida State Library, 2000) much work remains to be done to make outcomes assessment a practical, valid, and easy approach that will be embraced by the library community. Those developing such manuals need to work toward methodologies that are feasible, cost effective, and actionable with relative ease. It is unclear at this time how well outcomes assessment--as a method--can be demystified and integrated easily into an ongoing library evaluation program.

Local Judgment Calls

Missing from a number of discussions about the importance of outcomes and outcomes assessment is the discussion of what are the "correct," "appropriate," or "right" outcomes for a particular library setting. Thus, the research question here is: What factors within the library and the local community affect the judgment of when a service is "good enough"? The issue becomes more complicated if the library has to accommodate its outcomes with those of a larger institution (university or city government). Moreover, once there is a selection of a particular outcome as appropriate for that setting, "how much" of that outcome does the library need to obtain to claim "success"? Similar issues occur in other types of assessment methods. The concern, however, is in method replacing value judgments on the part of the users of these methods.

As an example, the performance measure "correct answer fill rate," offers no guidance unless we know if a 50 percent correct answer fill rate is acceptable for a particular library or if a 75 percent correct answer fill rate is desired instead. Having a performance measure, or having an outcome, is not the same as agreeing on the quality or performance level that a library wishes to achieve on that outcome or performance level. It is the responsibility of the individual library or organization to develop a process that results in agreement that the quality standard for correct answer fill rate might be something like, "reference staff will answer quick fact and bibliographic questions correctly 70 percent of the time."

The quality level or "standard" that a library or organization accepts as a benchmark by which to judge itself depends on:

* Institutional and library goals and objectives (either developed internally or imposed externally);

* Stated priorities among those objectives (libraries cannot be all things to all people all the time);

* Resource allocations among goals, objectives, and services;

* Staff knowledge and skills in providing information services; and

* Other situational factors at play at an individual setting.

The easy part of evaluation may be identifying and validating the performance measures, outcome, or various statistics as indicators of information services. The hard part may be the development of a process within the organization to agree on the quality standards for those outcomes that are acceptable, or are the target, for that particular organization. In short, outcomes assessment has little utility without accompanying quality standards, and these require judgment calls on the part of the users of such methods.

Recognizing the Political Context of Outcomes Assessment

Evaluation, overall, has significant political overtones beginning with the determination of what to evaluate, how it will be evaluated, what outcomes are most important, what measures to use, how to interpret findings, and how best to report or present the findings. Noticeably absent from a number of the discussions of outcomes assessment are considerations related to the political environment in which the library exists. A determination of "appropriate" outcomes by the larger institution can eliminate effectively the ability of the library to demonstrate how it supports those outcomes. In addition, these political considerations are compounded because of the network and technology infrastructure through which library services and resources are provided. For example, a public library that is dependent on a city's technology infrastructure for its network and resultant services has a more complicated political context for determining and measuring outcomes than a library that controls its own technology.

Dealing with the various political issues related to outcomes and outcomes assessment in a networked environment is new to most libraries. In addition, outcomes assessment in the highly charged technological infrastructure of many organizations requires the evaluator to have a range of skills and knowledge not needed in the nonnetworked environment. Research is needed to shed light on those serious issues.


As suggested in the key research questions posed earlier in this article, it is necessary to view research related to outcomes and outcomes assessment in the larger context of overall library and information services assessment. Outcomes assessment is but one method that has the potential to help library managers make better decisions regarding the provision of information services and resources. At present, however, there are many issues and problems with practical implementations of the approach. While there certainly is potential for developing outcomes assessment, much work--especially empirical research--is necessary.

As discussed previously, use of the terms outcomes, performance measures, quality standards, service quality, etc. continue to be disputed. As Kyrillidou (2002) concludes in a recent article, not only is there a need for more dynamic models to describe libraries and users, "we lack an adequate shared understanding of how outcomes are defined" (p. 45). The framework outlined in this article (see Figure 2) is one effort to help clarify the use of these terms and provide a means to understand their relationship to each other. Without better agreement among researchers about the use of these terms there is likely to be limited implementation of these various assessment techniques. Indeed, at a recent workshop conducted by the authors, a participant commented, after a review of the various assessment methods and approaches, that researchers and writers studying outcomes were like kids playing in a sandbox--and that clear, practical, usable evaluation methods and approaches are not available to practitioners.

Also lacking from many library evaluation approaches is adequate recognition of how the networked environment affects services and resource provision. As shown earlier in Figure 1, the way online services are accessed raises a range of evaluation issues that require substantial attention. A key theme throughout this article is that outcomes assessments, as well as other types of library assessment, have yet to address factors and issues arising from the networked environment that seriously complicate valid assessment approaches. Indeed, little of the outcomes assessment work to date considers the evaluation of network-based services or resources--and how those services/resources differ from traditional library services or resources. And yet, libraries are increasing the network-based services and resources that they provide.

There are a number of possible next steps for researchers working in the area of outcomes assessment:

* Develop a better understanding of outcomes and how outcomes "fit" into the range of evaluation models. This could be done by comparing the strengths and weaknesses of the models as shown in Table 2 and by reviewing the considerations given in this article that the models do not address currently.

* Specify and rethink college and university accreditation documents to better determine what they mean by outcomes assessment and the process by which such assessment should be done. The review of such accreditation documents by Gratch-Lindauer (2002) clearly identifies confusion and lack of understanding by these accreditation agencies concerning outcomes assessment.

* Involve library practitioners in research related to outcomes to obtain their knowledge, views, and experiences in evaluation.

* Conduct empirical research on the validity of Table 3 in this article and determine the degree to which outcomes are similar or different in a traditional (print and face-to-face service provision) environment versus a networked environment.

* Conduct empirical research that determines the extent to which outcomes are generic and, thus, obtainable and comparable across a range of libraries and library types based on library situational and other factors.

* Sponsor a small meeting or symposium in which key researchers and practitioners could debate and discuss a range of issues and concerns regarding outcomes assessment. Such a meeting could help focus attention on how best to attack outcomes assessment as an evaluation method.

These are only a few efforts possible to begin to address some of the research questions identified in this article. Until such steps are taken and a range of research activities initiated, outcomes assessment will continue to be an idea in search of both a theory and a practice.

There is much work yet to do in translating outcomes assessment and other evaluation approaches into practical, useful, and valid assessment techniques in the networked environment. The track record for the degree to which practitioners have embraced a program of ongoing evaluation (regardless of the method) is extremely low. Lakos (1999) identified the need for a "culture of assessment" within libraries years ago--and it is still lacking in large part. Needed now is more empirical research related to these evaluation approaches; evaluation methods that recognize and understand the role of the networked environment in the provision of services and resources; partnerships with libraries and related organizations to test, refine, and validate practical and doable approaches; and a commitment to developing multiple evaluation methods that can work successfully together.
Table 1. Selected Definitions of Outcomes and Outcomes Assessment. *

Citation Definition of Outcome

Zweizig, D., Johnson, D., Robbins, J., "Outcomes--how things
& Besant, J. (1994). TELL IT! changed for the
Evaluation sourcebook and training community" (p. 104).
manual. Madison, WI: University of
Wisconsin, Madison, School of
Library and Information

United Way. (1996). Measuring program "Outcomes are benefits for
outcomes: A practical approach. participants during or after
Alexandria, VA: United Way. their involvement with a
 program. Outcomes may relate to
 knowledge, skills, attitudes,
 values, behavior, condition, or
 status" (p. xv).

Himmel, E., & Wilson, W. J. (1998). "... a service response is what
Planning for results: A public a library does for, or offers
library transformation process: The to, the public in an effort to
how-to manual. Chicago: American meet specific community needs.
Library Association. Service responses represent the
 gathering and deployment of
 critical resources to produce a
 specific public benefit or
 result" (p. 51-52).

Himmel, E., & Wilson, W.J. Planning "Evaluation is the process
for results: A public library used to measure the
transformation process: The performance of a
guidebook, Chicago: American service against some
Library Association. pre-determined criterion
 to see how well or poorly
 the service has been
 performed" (p. 35).

Institute of Library and Museum Outcomes are "benefits or
Services. (2000). Perspectives on changes for individuals
outcome based evaluation for or populations during or
libraries and museums. after participating in
Washington, D.C.: Institute of program activities ..."
Library and Museum Services. that are measured against
 predetermined criteria
 (p. 20).

Free Library of Philadelphia. Outcomes are "the quality
(2000, November 30). Free Library of things produced" as
of Philadelphia: Performance related to the goals set
management final report. forth early in the project's
Philadelphia, PA: Price development (E, p. 1).
Waterhouse Coopers.

Association of College and Focus "on the achievement of
Research Libraries. (2000). outcomes that have been
Standards for college libraries identified as desirable in
2000 edition. Chicago: American the library's goals and
Library Association, ACRL objectives. It identifies
College Libraries Section performance measures,
Standards Committee. such as proficiencies,
Available at: http://www. that indicate how well the library is doing what it
 has stated it wishes to do."

Bertot, J. C., McClure, C. R., & "An outcome measure is
Ryan, J. (2001). Statistics and explicitly tied to the
performance measures for public libraries goals, objectives,
library networked services. and planning process. A
Chicago: American Library good outcome measure provides
Association. data that tells a library
 manager if a specific
 library objective
 has been achieved" (p. 66).

Hernon, P., & Dugan, R. E. (2002). "Outcomes assessment deals
An action plan for outcomes assessment with academic institutions
in your library. Chicago: providing evidence that they
American Library Association. are meeting their educational
 mission. In the case of the
 library, outcomes focus on how
 library users changed as a
 result of their contact with
 the library and its resources,
 services, and programs" (p. 2).

Fraser, B. T., & McClure, C. R. An outcome is a "clearly
(2002, forthcoming). Toward a identified result or end
framework for assessing product that occurs as a
library and institutional consequence of individual
outcomes. Portal: Libraries or combined activities from
and the Academy, 2(4), 505-528. units of the institution.
 It is a preferred or desired
 state and ideally clarifies
 specific expectations of
 what should be products
 from the institution."

* The authors wish to thank Lara Rudolph for her assistance in
compiling the citations in this table.

Table 2. Selected Outcome Models. *

Citation Model

Kyrillidou, M. (2002). From The Linear Model
input and output measures to
quality and outcome measure,
or, from the user in the life of The Cyclic Model
the library to the library in the
life of the user. The Journal of
Academic Librarianship, 28, pp. The Spiral Swirl

Bertot, J. C., & McClure, C. R. Outcomes and
(2003). Outcomes assessment in Performance
the networked environment: Measurement in
Research questions, issues, Libraries
considerations, and moving
forward. Library Trends 51:4.

Hernon, P. (2002). Outcomes Aggregate Factors
are key but not the whole Approach
story. The Journal of Academic
Librarianship, 28, pp. 54-55.

King et al. (2003). Library Derived Measures
economic measures. Library
Trends, 51 (3).

Fraser, B. T., & McClure, Preliminary
C. R. (2002). Toward a Framework of the
framework for assessing Outcomes Assessment
library and institutional Process
outcomes. Portal: Libraries
and the Academy, 2,
pp. 505-528.

Citation Summary

Kyrillidou, M. (2002). From Based on the "assumption
input and output measures to that inputs have a
quality and outcome measure, direct relation to
or, from the user in the life of outputs, which, in turn,
the library to the library in the relates to quality
life of the user. The Journal of and outcomes" (p. 44).
Academic Librarianship, 28, pp.
42-46. Acknowledges that user
 transactions have
 "multiple dimensions
 of inputs, output, and
 quality and outcome
 elements from multiple
 interactive and reflective
 perspectives" (p. 44-45).

 "[T]ries to introduce
 the notion of motion
 depicting a more
 dynamic and flexible
 model, moving users
 and information
 resources into a
 spiral swirl up and down
 into the depths of
 knowledge, exploration,
 and experience" (p. 45).

Bertot, J. C., & McClure, C.R. "Inputs and outputs form
(2003). Outcomes assessment in the basis for service
the networked environment: quality and outcomes
Research questions, issues, assessment activities.
considerations, and moving Outcomes assessment seeks
forward. Library Trends 51:4. to determine the impact of
 the library's
 (again, outputs) on the
 library service and
 resource users. One
 cannot have outcomes
 measures without
 measures of outputs--
 libraries need to know
 what investments (inputs)
 produce what services
 (outputs) in order to
 determine the perceived
 quality (quality assessment)
 and impacts (outcomes) of
 those services/resources"
 (p. X).

Hernon, P. (2002). Outcomes Sees "library planning and
are key but not the whole decision making as
story. The Journal of Academic revolving around service
Librarianship, 28, pp. 54-55. quality and its companion
 concept, satisfaction, and
 outcomes (or more precisely
 outcomes assessment). Inputs
 and outputs support the
 accomplishment of service,
 quality, satisfaction, and
 an assessment plan" (p. 54).

King et al. (2003). Library "Measures are designed to
economic measures. Library serve the perspectives of
Trends, 51 (3). library staff management,
 library users, the
 fundors of the library and
 the higher order community
 served by the library" (X).
 "Outcomes are best determined
 by relating them to the
 purposes for which
 the information is
 obtained ..." (X).

Fraser, B. T., & McClure, Model depicts "a basic
C.R. (2002). Toward a process by which an
framework for assessing academic research library
library and institutional helps meet the goals of
outcomes. Portal: Libraries particular departments and
and the Academy, 2, functional units within a
pp. 505-528. university, which in turn
 contribute to institutional
 goals, while acknowledging
 that libraries may also
 contribute more
 directly" (p. 519).

* The authors wish to thank Kim A. Thompson for her assistance
in compiling the citations in this table.

Table 3. Outcomes and Outcome Levels.

Outcome Outcome Level
Time Dimension User Library Institutional System State National



Association of Research Libraries. (2002). Measures for electronic resources (e-metrics). Unpublished announcement. Washington, D.C. Retrieved December 12, 2002, from http://

Bertot, J. C. (2000). Libraries on the information highway: Issues and lessons learned. In P.D. Fletcher & J. C. Bertot (Eds.), World libraries on the information superhighway (pp. 288-300). Hershey, PA: Idea Group Publishing.

Bertot, J. C. (2001). Measuring service quality in the networked environment: Approaches and considerations. Library Trends, 49(4), 758-775.

Bertot, J. C., McClure, C. R., & Davis, D. M. (2002) Developing a national data collection model for public library networked statistics and performance measures: Interim report. Washington, D.C.: Institute of Museum and Library Services.

Bertot, J. C., McClure, C. R., & Ryan, J. (2001). Statistics and performance measures for public library networked services. Chicago: American Library Association.

Cook, C., & Heath, F. M. (2001). Users' perceptions of library service quality: A LibQUAL+ qualitative study. Library Trends, 49(4), 538-584.

Cook, C., Heath, F. M., Thompson, B., & Thompson, R. L. (2001). The search for new measures: The ARL LibQUAL+ study--A preliminary report. Portal: Libraries and the Academy, 1(1), 103-112.

Dresang, E. T., & Gross, M. (2001). Evaluating children's resources and services in a networked environment. In C. R. McClure & J. C. Bertot, (Eds.), Evaluating networked information services: Techniques, policy, and issues (pp. 23-44). Medford, NJ: Information Today.

Florida State Library. (2000). Workbook: Outcome measurement of library programs. Tallahassee: Florida Department of State, Division of Library and Information Services.

Fraser, B., & McClure, C. R. (2002). Toward a framework for assessing library and institutional outcomes. Portal: Libraries and the Academy, 2(4), 505-528.

Gratch-Lindauer, B. (2002). Comparing the regional accrediation [sic] standards: Outcomes assessment and other trends. Journal of Academic Librarianship, 28(1-2), 14-25.

Hernon, P. (2002). Outcomes are key but not the whole story. Journal of Academic Librarianship, 28(1-2), 54-55.

Hernon, P., & Dugan, R. E. (2002). An action plan for outcomes assessment in your library. Chicago: American Library Association.

Hernon, P., & Whitman, J. R. (2001). Delivering satisfaction and service quality: A customer-based approach for libraries. Chicago: American Library Association.

Institute of Museum and Library Services (IMLS). (2000). Perspectives on outcome based evaluation for libraries and museums. Washington, D.C.: Institute of Museum and Library Services. Retrieved December 4, 2002, from <>.

King, D. W., Boyce, P., Montgomery, C. H., & Tenopir. C. (2003). Library economic measures: Examples of the comparison of electronic & print journal collections and collection services. Library Trends, 51(3), 376-400.

Kyrillidou, M. (2002). From input and output measures to quality and outcome measures, or from the user in the life of the library to the library in the life of the user. Journal of Academic Librarianship, 28(1-2), 42-46.

Lakos, A. (1999). The missing ingredient: Culture of assessment in libraries. Performance Measurement and Metrics, 1(1), 3-7. Retrieved March 20, 2003, from

Lance, K. C., Steffen, N. O., Logan, R., Rodney, M. J., & Kaller, S. (2002). Counting on results: New tools for outcomes-based evaluation of public libraries. Aurora, CO: Bibliographical Center for Research. Retrieved January 6, 2003, from CountingOnResults.htm.

McClure, C. R., & Bertot, J. C. (1998). Public library use in Pennsylvania: Identifying uses, benefits, and impacts. Harrisburg, PA: Office of Commonwealth Libraries, Department of Education.

McClure, C. R., & Bertot, J. C. (Eds.). (2001). Evaluating networked information services: Techniques, policy, and issues. Medford, NJ: Information Today.

McClure, C. R., Fraser, B. T., Shim, W., & Bertot, J. C. (2002). Identifying and measuring library activities/services related to academic institutional outcomes. Washington, D.C.: Association of Research Libraries.

McClure, C. R., Lankes, R. D., Gross, M., & Choltco-Devlin, B. (2002). Statistics, measures, and quality standards for assessing digital reference library services: Guidelines and procedures. Syracuse, NY: Information Institute of Syracuse.

Shim, W.J., & McClure, C. R. (2002). Improving database vendors' usage statistics reporting through collaboration between libraries and vendors. College & Research Libraries, 63(6), 1-16.

Shim, W.J., McClure, C. R., Fraser, B. T., & Bertot, J. C. (2002). Data collection manual for academic and research library network statistics and performance measures. Washington, D.C.: Association of Research Libraries.

Smith, K. R. (2000). New roles and responsibilities for the university library: Advancing student learning through outcomes assessment. Washington, D.C.: Association of Research Libraries. Retrieved November 8, 2002, from

Smith, M., & Rowland, G. (1997, May-June). To boldly go: Searching for output measures for electronic services, Public Libraries, 168-172.

United Way of America. (1996). Measuring program outcomes: A practical approach. Alexandria, VA: United Way of America.

Van House, N., Lynch, M.J., McClure, C. R., Zweizig, D. L., & Rodger, E. J. (1987). Output measures for public libraries: A manual of standardized procedures. Chicago: American Library Association.

John Carlo Bertot, Associate Professor and Associate Director, and Charles R. McClure, Francis Eppes Professor and Director, Information Use Management Policy Institute, School of Information Studies, 242 Louis Shores Building, Florida State University, Tallahassee, FL 32306-2100

JOHN CARLO BERTOT is Associate Professor and Associate Director of the Information Use Management and Policy Institute in the School of Information Studies at Florida State University. He teaches courses in library technology planning, technology applications, and information and telecommunications policy. Bertot has published extensively in the areas of library management, planning, and evaluation, with a particular emphasis on library Internet use and involvement. Most recently, he coauthored Statistics and Performance Measures for Public Library Networked Services through the American Library Association (2001), and coedited Evaluating Networked Information Services, an Information Today publication (2002). At present, Bertot is co-principal investigator for a national U.S. study sponsored by the Institute of Museum and Libraries to develop a national model for the collection and reporting of public library network statistics and performance measures as well as for the development of training modules to assist libraries in collecting, using, and reporting network-services data. Bertot also serves as a U.S. delegate on the International Standards Organization's (ISO) Library Statistics committee, and is the convener of the ISO Library Performance Indicator committee. He is also a member of the National Information Standards Organization's (NISO) planning committee for the revision of the Z39.7 Library Statistics standard.

CHARLES R. MCCLURE, who obtained his Ph.D. from Rutgers University, is the Francis Eppes Professor of Information Studies at the School of Information Studies, Florida State University. He is also the Director of the Information Use Management and Policy Institute at Florida State University. McClure was the co-principal investigator on the Association of Research Libraries' funded E-metrics project, 2000-01; he was the co-principal investigator on a study funded by the U.S. Institute for Museum and Library Services to develop and test measures for networked services, 2000-02; and he recently completed a study funded by OCLC and a consortium of other library organizations that resulted in a manual to assess digital reference services. He is the coauthor of Statistics and Performance Measures for Public Library Networked Services (American Library Association, 2001); Evaluating Networked Information Services (Information Today, 2002); and Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services (Syracuse University Press, 2002).
COPYRIGHT 2003 University of Illinois at Urbana-Champaign
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2003, Gale Group. All rights reserved. Gale Group is a Thomson Corporation Company.

Article Details
Printer friendly Cite/link Email Feedback
Author:McClure, Charles R.
Publication:Library Trends
Date:Mar 22, 2003
Previous Article:Public library service to children and teens: a research agenda.
Next Article:Electronic publishing: research issues for academic librarians and users.

Related Articles
An emergency department perspective: outcomes.
Principles and a case study.
Where the environment fits in public health.
The Regulatory Dilemma.
Getting Real: Implementing General Education Assessment that Works.
World libraries on the information superhighway: Internet-based library services.
Reflection through the ID-PRISM: a teacher planning tool to transform classrooms into web-enhanced learning environments.
Lessons learned for the National Children's Study from the National Institute of Environmental Health Sciences/U.S. Environmental Protection Agency...

Terms of use | Copyright © 2017 Farlex, Inc. | Feedback | For webmasters