Printer Friendly

Measuring the value and impact of health sciences libraries: planning an update and replication of the Rochester Study.

DOI: 10.3163/1536-5050.97.4.016


In 2007, the National Network of Libraries of Medicine (NN/LM), Middle Atlantic Region (MAR), formed a planning group to explore the possibility of replicating a landmark study on the value of hospital libraries and their impact on clinical care, popularly known as "the Rochester Study" [1]. The Rochester study was among the first studies to relate information services provided by librarians to patient care outcomes, and it continues to be cited as evidence of the value of library services. The purpose of this paper is to update the library community on the progress of the proposed value of libraries study.


The impetus for the 1992 Rochester study was a decision by the state of New York that it would not continue to require hospitals to have a library because there was no evidence that having a library made any direct contribution to improved patient care. The librarians in the Rochester area took up the challenge and sponsored a study to explore the value and impact of the information provided by the hospital library, building on an earlier study in Chicago [2]. While the Rochester study was not a randomized controlled trial, it did incorporate a number of rigorous study design characteristics that increased the credibility of the results. The study used a critical incident technique [3], in which physicians and residents were asked to request information from their hospital librarian related to a current clinical case and to complete a questionnaire focusing on the impact of that information on their clinical decision making. The participants were randomly chosen; their names were not provided to the librarians; the study was prospective as opposed to retrospective; and all fifteen hospital libraries in the five-county area around Rochester, New York, took part.

The research questions in the Rochester study were: (1) whether hospital libraries provided information services that were perceived as valuable by physicians and (2) whether the received information had an impact on patient care. Physicians were seen as the key decision makers in the health care system and in the best position to judge the quality of information and its impact on patient care. The Rochester study assessed the quality of the provided information, the cognitive value of the information, its contribution to patient care, and time savings generated by receiving the information. The study identified specific impacts or changes in key decisions made by physicians in response to received information, such as diagnosis, choice of tests, and choice of treatment, as well as avoidance of adverse patient events.

The Rochester study and its Chicago predecessor were among the first studies to move beyond measuring basic inputs, outputs, and even outcomes to measuring perceived value and impact of information. It demonstrated that in the eyes of the information users (in this case, physicians) that library services were valued and that the provided information was seen as making a positive difference in patient care. The Rochester study has been heavily cited, achieving a prominent influence in the field, not only among librarians, but also in the medical literature [4, 5].


In the years following the Rochester study, a number of other studies examined the impact of library services, the effect of a clinical medical librarian, or the effect of readily available or point-of-care electronic resources for physician use. These studies often incorporated the Rochester study patient care outcome measures, such as improved diagnosis, choice of tests, choice of therapy, and reduced length of stay [4, 5]. Usage studies are particularly well suited to valuing services that are normally provided at no charge to users.

Two recent systematic reviews of the literature on the value of the impact of library services on patient care [6, 7] found that there were studies of sufficient rigor and quality to serve as a sound foundation of evidence that information provided by a librarian, whether through mediated search and reference services or through the selection and provision of electronic resources, had a positive impact. Weightman and Williamson's [6] comprehensive review of the literature on the impact of information provided through library services for patient care found 28 studies with sufficient rigor to provide valid measures of impacts of library-provided information on patient care, diagnosis, choice of tests, choice of therapy, and reduced length of stay. The most commonly reported impacts were a general impact on clinical care (37%-97% of respondents reporting in studies reviewed), advice to patients (47%-72%), and diagnosis and treatment or management (25%-61%). Bryant and Gray [7] covered ground similar to that of Weightman and Williamson, though they noted the paucity of research focusing on the value of library-provided information in primary care.


There is conflicting evidence on the effectiveness of the clinical medical librarian on patient care outcomes. As recently as 2003, one literature review has found scant evidence supporting the effectiveness of the clinical medical librarian [8]. Studies have emphasized the role of the librarian, interaction with the health care professional, and integration in the clinical setting [9, 10] rather than measuring actual impact. However, the clinical informationist model has matured [11], and a growing body of evidence supports the effectiveness of clinical information services for patient care. Recent evidence suggests that physicians who receive clinical information services are more likely to try a new or different treatment than physicians who do not receive clinical information services [12], and librarians' presence at morning report, followed by a literature review, can reduce the length of stay of patients in a hospital [13].


A number of studies have assessed the impact of MEDLINE searches [14] and point-of-care resources [15]. A recent study at the University of Illinois created an "academic return on investment (ROI)," assessing the impact of the availability and provision of library resources on university-obtained grants [16]. A monetary assessment was assigned to a set of services through a formula determined by the researchers. Many such studies, though, do not make the distinction between library-provided and nonlibrary-provided resources, and occasionally the method of obtaining the information (the "Internet") may be put in the same category as the content (continuing medical education lecture) [17].


The NN/LM, MidContinental Region (MCR), is working on value of libraries studies throughout NN/LM MCR (Colorado, Kansas, Nebraska, Missouri, Utah, and Wyoming). Led by the J. Otto Lottes Health Sciences Library at the University of Missouri--Columbia, the research project will survey library users to establish how they use the library and how the library-provided information supports their patient care, teaching, and research. It will be implemented at three hospital libraries in Missouri, the health sciences library at the University of Colorado, and three hospital libraries in Colorado.

In addition to conducting this study, the NN/LM MCR has modified tools for health sciences library use: a valuing library services calculator [18], adapted from the Massachusetts Library Association and then adapted for the web by Chelmsford Public Library, that provides a monetary assessment of the worth of a library to an institution. Using a web-based form, librarians can enter the number of uses of a resource, multiplied by the estimated cost of providing the resources, to determine a value of the resource relative to its use. Another tool offered on site is a cost benefit and ROI calculator [19], which librarians can use to assess the benefit to the institution for the money spent on library resources.

Several independent researchers are actively involved in studies of the value of libraries. Beth Hill, AHIP, of the Kootenai Medical Center in Idaho, a member of the Task Force on Vital Pathways for Hospital Librarians and a doctoral candidate in education, is implementing a replication of the Rochester study in rural (twenty-five or fewer beds) critical access hospitals (CAHs) in the states of Alaska, Idaho, Montana, and Washington. One hundred CAHs have thus far agreed to distribute the survey to their active staff of physicians, physician assistants, and nurse practitioners. Christine Urquhart of the University of Wales, author of many studies on the value of libraries, is working with Alison Weightman in developing and refining the "United Kingdom Value of Libraries Toolkit" [20].


The limitations of many of the value of libraries' studies are that often the "n" in most studies is small, leading to difficulties in generalizing the results and that, with many studies taking place at a single institution, the confounding factors can be the librarians providing the information service and the specific nature of the population of patient care providers and patients at the institution. Studies of the value of libraries or librarians also focus on narrow aspects of the provision of information services: for example, impact of information services on one specific patient outcome, length of stay [13], or the impact of a clinical librarianship service on patient care [12]. These studies generally do not examine the impact of library-provided print and electronic resources on patient care.


One of the challenges facing the planning group is replicating the Rochester study in the current information and library services environment. The current environment is much more complex than that of the mid-1980s. There are more opportunities for users to access information resources on their own, without requiring the services of a librarian. The technological barriers to using the resources have been reduced, allowing even untrained users to successfully navigate electronic resources and locate needed information using systems that formerly required extensive training to search effectively. The librarian's skills in mediated searching may still be required for complicated or time-consuming information requests; however, the essential expertise librarians bring to selecting and making resources available to users has become more complex and yet more transparent in an electronic environment.

The planning group agreed that understanding both hospital administrators' and hospital librarians' points of view would be helpful in designing the study. Focus groups of librarians who responded to a general call to participate and who had previously interviewed their hospital administrators using questions developed by the planning group (Table 1) were conducted by an outside consultant.

Responses common to both of the focus groups were that the administrators listened to and found valuable the opinions and needs of their medical staff, nurses, and patients. The library was still valued not only for its physical space, but also for its resources and role in supporting patient care, education, and administrative decision making. Suggestions from the administrators as to how the library could additionally demonstrate value included connecting evidence to bedside care of the patient, possibly through linking library resources to the electronic medical record. The measurement of value for administrators remained quantitative: library usage numbers, for instance, were highly valued [21].


Building on information from the focus groups on the library services that key stakeholders value, the proposed study will examine the impact, or value, of libraries and librarians on the clinical decision making of physicians, residents, and nurses. The study will use "triangulation," the "weaving together of different data gathering techniques ... to help ensure that the resulting descriptions and interpretations are as useful as they can be" [22]. Three different but overlapping aspects of the "value of library" will be measured (Table 2). The first two--the value of the information itself and the value of the librarian--will be assessed for clinical providers through an email survey and semi-structured interviews. The third--the attitudes and insights of the librarians--will be studied in focus groups and analyzed by a facilitator.

The survey for the new study will be similar in many ways to that used in the Rochester Study, although it will be retrospective rather than prospective. The respondents will be asked to "think of one occasion during the past month when you needed information related to patient care, and answer the questions based on that occasion." These questions relate to the relevancy and usefulness of the information:

* Was the information relevant and current?

* Did it refresh my memory or provide new knowledge?

* Did it result in better-informed clinical decisions or higher quality of care?

* Did it save me time?

The survey will also inquire about any cognitive or behavioral changes that the information may have brought about: change in diagnosis, choice of tests, choice of drugs, and so on, or avoidance of hospital admission, surgery, hospital-acquired infection, and so on. Provider interviews in the second prong of the study will be designed to focus specifically on the impact of the librarian in the provision of information services. The third component of the study, librarian focus groups, will gain insights from health sciences librarians in order to provide a clearer picture of the information-seeking process and its value. The study will come full circle, from the initial librarian focus groups (completed in fa112007) to the perceptions and observations of librarians and their reactions to the survey and interview results. As the last component of this triangulated research design, the librarian focus groups will help the study group compare and contrast data from the varying sources, thereby verifying and strengthening the results that emerge.


A number of challenges remain for the proposed study. Best practice for survey research suggests that incentives are important to increase response rate [20], but with participants spread out across many hospitals and the only contact with them via email, offering incentives is complicated. In addition, some public institutions are not permitted to offer incentives to research participants. The planning group has also discussed whether a second phase of the survey might include hospital libraries that do not initially volunteer to participate. To alleviate bias that may occur from including volunteer libraries only, the planning group may approach some non-volunteer hospital libraries and offer to assist them in conducting the survey at their institutions. Providing such assistance would widen the group of participating libraries. Nonresponse studies conducted at the institutional participation and respondent levels are also a possibility.

Pretesting the survey instrument and conducting a pilot in a few institutions are the next steps for the group to move forward toward the launch of the full research study. There have been and continue to be many uncertainties on this research journey, but the planning group looks forward to the NN/LM MAR value of libraries study adding significantly to the growing body of literature that demonstrates the value of libraries and librarians to clinical decision making.

Like the original Rochester study, the planning phase of the NN/LM MAR study makes use of a community-based participatory research (CPBR) approach. Israel et al. describe CPBR as "a collaborative approach to research that equitably involves all partners in the research process and recognizes the unique strengths that each brings. CPBR begins with a research topic of importance to the community with the aim of combining knowledge with action and achieving social change" [23]. The social change goal referred to in this definition reflects the extensive use of CBPR in the public health and health promotion fields. The social change envisioned by the planning group involves creating a culture of assessment in which the value and impact of health information and libraries are understood and appreciated. CPBR has been a feature of the study from the initial focus groups of practitioners and throughout the planning process. The planning group consists of a mix of library practitioners and researchers who are working together to design a new value study that is both rigorous and rooted in the real world of library practice.


The Rochester study [1] is an influential study on the value of health sciences library services. Since it was published in 1992, it has been heavily cited and used by other studies of library value. Using the Rochester study as a base, the proposed value of libraries study will use a three-pronged approach to assess the value of libraries in a four-state area. The three prongs are a survey of physicians and nurses at hospital and health sciences centers on their use of print or electronic resources for patient care; interviews with selected physicians and nurses at hospital and health sciences centers on their use of librarian-mediated services (search, extended reference, filtering, and summarization) for patient care; and focus groups of librarians to review and confirm the findings from the survey and interviews. The proposed study is expected to demonstrate the value of library resources and librarian-provided services on patient care outcomes.

Submitted April 2009; accepted June 2009


Additional members of the Value of Libraries Planning Group who work, and continue to work, to make this "Value of Libraries Planning Study" emerge into a viable research study are: Susan Cavanaugh, Reuben Sharp Library, Cooper Hospital, University of Medicine and Dentistry New Jersey--Camden; Sharon Easterby-Gannett, AHIP, Lewis B Finn Library, Christiana Care Health System, Newark, DE; Mary Lou Klem, Health Sciences Library System, University of Pittsburgh Medical Center, Pittsburgh, PA; and Lynn Kasner Morgan, Levy Library, Mt. Sinai School of Medicine, New York, NY.


[1.] Marshall JG. The impact of the hospital library on clinical decision making: the Rochester study. Bull Med Libr Assoc. 1992 Apr;80(2):169-78.

[2.] King DN. The contribution of hospital library information services to clinical care: a study in eight hospitals. Bull Med Libr Assoc. 1987 Oct;75(4):291-301.

[3.] Flanagan JC. The critical incident technique. Psychol Bull. 1954 Ju1;51(4):327 58.

[4.] O'Connor P. Determining the impact of health library services on patient care: a review of the literature. Health Info Libr J. 2002 Mar;19(1):1-13.

[5.] Sherwill-Navarro PJ, Wallace AL. Research on the value of medical library services: does it make an impact in the health care literature? J Med Libr Assoc. 2004 Jan; 92(1):34-45.

[6.] Weightman AL, Williamson J, Library & Knowledge Development Network (LKDN), Quality and Statistics Group. The value and impact of information provided through library services for patient care: a systematic review. Health Info Libr J. 2005 Mar;22(1):4-25.

[7.] Bryant SL, Gray A. Demonstrating the positive impact of information support on patient care in primary care: a rapid literature review. Health Info Libr J. 2006 Jun;23(2): 118-25.

[8.] Winning MA, Beverley CA. Clinical librarianship: a systematic review of the literature. Health Info Libr J. 2003 Jun;20 suppl 1:10-21.

[9.] Urquhart C, Turner J, Durbin J, Ryan J. Changes in information behavior in clinical teams after introduction of a clinical librarian service. J Med Libr Assoc. 2007 Jan; 95(1):14-22.

[10.] Tod AM, Bond B, Leonard N, Gilsenan IJ, Palfreyman S. Exploring the contribution of the clinical librarian to facilitating evidence-based nursing. J Clin Nurs. 2007 Apr;16(4):621-9

[11.] Giuse NB, Koonce TY, Jerome RN, Cahall M, Sathe NA, Williams A. Evolution of a mature clinical informationist model. J Am Med Inform Assoc. 2005 May-Jun;12(3):249 55.

[12.] Mulvaney SA, Bickman L, Giuse NB, Lambert EW, Sathe NA, Jerome RN. A randomized effectiveness trial of a clinical informatics consult service: impact on evidence-based decision-making and knowledge implementation. J Am Med Inform Assoc. 2008 Mar-Apr;15(2):203-11.

[13.] Banks DE, Shi R, Timm DF, Christopher KA, Duggar DC, Comegys M, McLarty J. Decreased hospital length of stay associated with presentation of cases at morning report with librarian support. J Med Libr Assoc. 2007 Oct;95(4):381-7.

[14.] Lindberg DA, Siegel ER, Rapp BA, Wallingford KT, Wilson SR. Use of MEDLINE by physicians for clinical problem solving. JAMA. 1993 Jun 23-30;269(24):3124-9.

[15.] McCord G, Smucker WD, Selius BA, Hannan S, Davidson E, Schrop SL, Rao V, Albrecht P. Answering questions at the point of care: do residents practice EBM or manage information sources? Acad Med. 2007 Mar; 82(3):298-303.

[16.] Luther J. University investment in the library: what's the return? a case study at the University of Illinois at Urbana--Champaign [Internet]. Elsevier Library Connect, white paper #1 [cited 22 Mar 2009] <http://libraryconnect>.

[17.] Bennett NL, Casebeer LL, Kristofco RE, Strasser SM. Physicians' Internet information-seeking behaviors. J Contin Educ Health Prof. 2004 Winter;24(1):31-8.

[18.] National Network of Libraries of Medicine, MidContinental Region. Valuing library services calculator [Internet]. Salt Lake City, UT: The Network; 2009 [cited 19 Mar 2009]. <http:/ />.

[19.] National Network of Libraries of Medicine, MidContinental Region. Cost benefit and ROI calculator [Internet]. Salt Lake City, UT: The Network; 2009 [cited 19 Mar 2009]. <http:/ />.

[20.] Weightman A, Urquhart C, Spink S, Thomas R, on behalf of the National Library for Health Library Services Development Group. The value and impact of information provided through library services for patient care: developing guidance for best practice. Health Info Libr J. 2009 Mar;26(1):63-71.

[21.] Martin ER. Value of the hospital library: focus group report [Internet]. New York, NY: National Network of Libraries of Medicine, Middle Atlantic Region; 2008 [cited 19 Mar 2009]. < valuereport012208.pdf>.

[22.] Friedman CP, Wyatt JC. Evaluation methods in biomedical informatics. 2nd ed. New York, NY: Springer; 2006. p. 275.

[23.] Israel BA, Eng E, Schulz AJ, Parker EA. Methods in community-based participatory research for health. San Francisco, CA: Jossey-Bass; 2005.


Kathel Dunn, MSLS (corresponding author), Kathel, Associate Fellowship Coordinator, National Library of Medicine, Rockville Pike, Building 38, 2N-19, Bethesda, MD 20894; Karen Brewer, PhD, AHIP, FMLA,, Director, Ehrman Medical Library, New York University Health Sciences Libraries, 550 First Avenue, New York, NY 10016; Joanne Gard Marshall, MLS, PhD, FMLA,, Alumni Distinguished Professor, School of Information and Library Science, University of North Carolina at Chapel Hill, 100 Manning Hall, CB# 3360, Chapel Hill, NC 27599-3360; Julia Sollenberger, MLS, AHIP, FMLA, Julia_, Director, Health Science Libraries and Technologies, and Associate Professor, Department of Community and Preventive Medicine, University of Rochester Medical Center, 601 Elmwood Avenue, Rochester, NY 14642

* The project activities described in this paper have been funded by the National Library of Medicine, National Institutes of Health, Department of Health and Human Services, under contract no. N01-LM-6-3501, under the New York University School of Medicine Library.
Table 1
Questions used in focus groups of librarians, fall 2007

1. How are competing budgetary needs ranked and prioritized? Do key
individuals have a louder voice?

2. How much budgetary decision making is driven by compliance or
regulation? Can you provide an example?

3. Are there one or two specific things the library offers that are
especially useful to this organization?

4. Are there one or two specific services or resources that are
especially useful to you personally at work? When you need
information for your work, what is your usual approach to finding

5. Is there a challenge or opportunity for your organization where
the library could be involved? (Examples: performance improvement
initiatives, or length of stay, or patient satisfaction)

6. Does your organization involve your librarians in strategic
planning and/or hospital-wide, mission-critical committees? If not,
why not? What might make the librarian more central? What would
enhance the librarian's value to such committees?

7. What would convince you that the library is an essential resource,
worthy of appropriate funding? Can you think of specific measures of
library value that would be convincing to you?

8. Is there anything else you would want to say about libraries and
librarians that would help assess the value of these resources?

Questions asked of both administrators and librarians.

Table 2
Three components of the proposed National Network of Libraries
Medicine (NN/LM), Middle Atlantic Region (MAR), value study

To be assessed   Method              Library/institutional

Value of         Web-based survey    All hospitals and
  information                          academic health
  (especially                          institutions in the
  electronic                           the Middle Atlantic
  resources)                           Region (MAR) region
                                       invited to
Value of         Telephone           Selected institutions,
  librarian        interviews          representing range
                   semi-structured     of types of
                                       libraries and
Librarian        Focus groups        Range of types of
  attitudes,                           libraries and
  insights                             institutions

To be assessed   Individual participants

Value of         All physicians,
  information      residents, and nurses
  (especially      in participating
  electronic       hospitals and academic
  resources)       health institutions
                   will receive survey
Value of         20-30 total; 4 at each
  librarian        institution: 2 users of
                   searches, 2 users of
                   e-resources provided by
Librarian        Librarians representing
  attitudes,       range of types of
  insights         libraries and
COPYRIGHT 2009 Medical Library Association
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2009 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Dunn, Kathel; Brewer, Karen; Marshall, Joanne Gard; Sollenberger, Julia
Publication:Journal of the Medical Library Association
Article Type:Report
Geographic Code:1USA
Date:Oct 1, 2009
Previous Article:The magnet journey: opportunities for librarians to partner with nurses.
Next Article:Comparison of journal title coverage between CINAHL and Scopus.

Terms of use | Privacy policy | Copyright © 2020 Farlex, Inc. | Feedback | For webmasters