Systematic reviews and librarians.

ABSTRACT

Systematic reviews are review articles that are completed using predefined methods to minimize bias inherent to observational studies. Systematic reviews are important to librarians because they integrate evidence across studies or data resources to provide knowledge that is useful to good decision making in our profession. In addition, as more systematic reviews are being published in many disciplines, librarians are being asked to assist with the production of them--comprehensive searching is vital to the strength of the reviews. This article describes the process of producing systematic reviews and also describes their use. Librarians can acquire the skills necessary to use and produce high-quality systematic reviews.

INTRODUCTION

This article is designed to introduce librarians, both practitioners and researchers, to systematic reviews. I plan to set the context of this article by describing a scenario, define what a systematic review is in relation to all review articles, briefly discuss the history of systematic reviews, and list why they are important to librarians and why they are done. I will also describe the research strengths and quality indicators, show the steps in the production of a systematic review, discuss how one can find them across databases, and resolve the scenario. I include examples throughout the article from the disciplines of library and information science (LIS) as well as health care, the area of librarianship in which I have spent the past twenty years. The examples in this article are ones that I chose to show a specific aspect of systematic review production or use and represent a range of quality and content.

SCENARIO

You have just been promoted to be the head of one of the smaller inner-city branches of your local public library. One of the reasons you got the job is that you stressed the need for evidence from sound research to back up your library's services and collections. Your branch has a long and strong tradition of malay programs and is especially proud of their bibliotherapy programs run in conjunction with the local Department of Public Health. Your library director has just called and asked you to provide evidence concerning the effectiveness of the bibliotherapy programs. You sit down at your terminal and quickly find 252 articles in the National Library of Medicine's (NLM) PUBMED database. You sigh and wish that someone else besides you could "pull" all of these papers together and come up with a sound, evidence-based bottom line for bibliotherapy.

While you are thinking, the phone rings again and it is the director of the Department of Public Health. The city has just realized that their teen pregnancy rate is well above national and state levels and has started to push the Public Health people to "do something." The director, a long-time supporter of your services, asks if you could spare one of your librarians to help them search for and collect literature on prevention of teen pregnancy. The Health Department needs to write a report summarizing the evidence on the effectiveness of various approaches to preventing teen pregnancy and plan for new programs. Again you are faced with compiling the information on a certain topic or area and having it ready for others to apply--in other words, a systematic review of the literature.

WHAT IS A SYSTEMATIC REVIEW?

The research world recognizes two sorts of review articles, both of which are important. Narrative reviews are opinion pieces done by an expert in the field. They are often broad based, written by a single author, and lack formal summaries of whole bodies of knowledge. Narrative reviews provide valuable coverage of an area of knowledge or an introduction to a topic, similar to what would be found in a textbook chapter. Two useful examples of narrative reviews in L1S include bibliotherapy and bullying (Gregory & Vessey, 2004) and public libraries and ethnic minority communities in the UK (Elliott, 1999). Both were written by experts who summarized the content area of their respective topics. Students and those interested in a general summary of a topic value these expert (narrative) reviews.

Systematic reviews, on the other hand, are often much more narrowly focused and are written by a team of researchers who represent a range of skills and interest in the topic. Cook, Mulrow, and Haynes (1997) define them as reviews that assemble, critically appraise or evaluate, and synthesize the results of primary studies in an integrative approach. They continue by listing the features of a well-done systematic review:

1. It defines the question to be addressed precisely and explicitly

2. It includes a replicable search strategy (for example, databases, terms, years, language restrictions, and other limits)

3. It uses pre-set inclusion and exclusion criteria to select articles or data sources that will be summarized in the review

To illustrate the features of a systematic review, it is worth looking at a study by Weightman and Williamson (2005). These authors wanted to examine the research on the value and impact of information about patient care provided by health sciences librarians. Their goal was "to review studies looking at the value and impact of library services on health outcomes for patients and time saved by health professionals" (p. 5).

They searched six databases (Education Resources Information Center [ERIC], PUBMED, Library and Information Science Abstracts [LISA], PREMDELINE, EMBASE[c], and Cochrane Database of Systematic Reviews[c]) using twenty terms in various "and" and "or" combinations. They also searched Google, did a hand search of two journals, and checked all bibliographies of the articles they retrieved. In addition, they contacted authors, used personal reprint collections, consulted peers, and submitted emails to several discussion groups and listservs. Their inclusion and exclusion criteria for individual studies spanned several paragraphs. Starting with 320 papers, they reduced the total to 68 papers on the first screen for inclusion and, with closer examination and data extraction, reduced the papers to the final 28 studies that were analyzed in their systematic review.

Their systematic review is well done and provides a strong base for building and maintaining professional library services for patients and health care providers. The bottom line across twenty-eight studies is that professionally led library services do impact health outcomes for patients and save time tot health care staff.

Systematic reviews can be qualitative or quantitative in nature. The former combines the information from the studies and describes results in a verbal format (for example, "four studies of clinical librarian projects suggested that professionals saved time"; "two studies showed evidence of cost-effectiveness"). The data from quantitative studies can, but not necessarily, be combined numerically and statistically. If this type of numerical and statistical combining is done (see, for example, Anderson et al., 2005, who looked at self-help books for depression), the systematic review is also classified as a meta-analysis.

HISTORY OF SYSTEMATIC REVIEWS

Systematic reviews date back to 1904 (Pearson, 1904). Although the first few were in medicine, many researchers and clinicians in education and psychology, as well as other social sciences disciplines, have done much to develop and improve systematic review methods and reporting. Much of this early work was concentrated in the 1960s and 1970s (see, for example, Glass, 1976). Systematic reviews are becoming more common in the health sciences, where researchers have built upon the evidence-based practice movement. Many other disciplines are producing more systematic reviews, in part because of the ease of finding studies and data sources to combine as well as advancement of systematic review methods. For anyone interested in learning more about systematic reviews, a very readable work describing the systematic review process in health care is available on the Internet from the Millbank Memorial Fund (Moynihan, 2004). Many discipline-specific texts also exist on systematic review and meta-analysis production.

WHY ARE SYSTEMATIC REVIEWS IMPORTANT TO LIBRARIANS AND LIBRARIANSHIP?

Systematic reviews and meta-analyses are important to librarians for two main reasons. First, they help us build and make sense of our own research base. Using systematic reviews we can more easily identity our strengths while finding out where gaps exist. Using information from systematic reviews, we can implement and justify valuable services and programs while stopping or bypassing those programs that have not been shown to be beneficial. A well-done systematic review means that individuals do not have to collect and analyze primary studies for every decision they make.

Second, we are the professionals who have access to and who can effectively access the world's knowledge. Systematic reviews and recta-analyses and their results are only as strong as the evidence that is gathered for analysis. Librarians have been key players in many systematic reviews. The role includes locating published reviews and identifying and obtaining studies for new systematic reviews. Harris (2005), a health sciences librarian who has done considerable work in the field, summarizes the role of an information scientist in the systematic review process. She outlines many considerations for anyone interested in being a team member on a systematic review project and the roles that a professional librarian can take in the process. Because systematic reviews are important to librarians, I feel that it is important for us to know their strengths and weaknesses as well as understand the production process so that we can not only use them effectively but also assist in their production.

WHY ARE THEY DONE?

Systematic reviews are done for many academic and application-based reasons. They are useful in the following cases.

* Too much information is available. For example, Ondrusek (2004) studied the attributes of research on end-user behavior for both online catalog and document retrieval systems--an almost insurmountable task. Her final report (45 pages) brings together 163 studies published in 175 articles. Analyses were done on year of publication, research populations, research methodologies (qualitative and quantitative), performance analyses (outcomes and obstacles), factors affecting performance (tasks, systems, and end-user traits), and historical trends. She not only summarizes her findings in tables and narrative form but also talks about the implications of her findings for librarians who make decisions about search engines for Web sites. Anyone interested in end-user searching would save much time by reading her study or working from the bibliography.

* Too little information is available. Some issues occur so infrequently that to understand them fully and systematically means going to previous literature. Even case reports of single episodes can provide integration and knowledge. This type of systematic review of uncommon occurrences is more common in health care than in some other disciplines. For example, scoliosis (curvature of the spine that can sometimes be fatal) is thought to be a genetic disorder. To remove some of the environmental and other biases in research into the causes of scoliosis Kesling and Reinker (1997) sought to study twins with scoliosis. Rather than waiting for occurrences of twins with the disorder to happen, the authors went to the literature. They found published data on 100 cases of sets of twins with scoliosis at adolescence. Sufficient data for analysis were available for 68 sets of twins. Analysis showed that genetics does play a strong role in adolescent idiopathic scoliosis.

* To resolve discrepancies. Systematic reviews can often (but not always) help uncover the truth about hotly debated issues. A recent systematic review on the effectiveness vitamin C in preventing the common cold comes to the conclusion that despite thirty or more years of controversy, vitamin C does not protect against colds. The review carefully lays out the history of the issue of benefit from vitamin C, including publication of two systematic reviews that came to very different conclusions (Douglas et al., 2006).

* To plan for new research. Research need is one of the most important drivers of systematic review production. Established researchers or those with a well-defined research direction produce systematic reviews to justify and plan future work, build on the work of others, communicate their findings, and position their research ideas in the field. Reading systematic reviews produced by other researchers and practitioners, especially the background, conclusions, and discussion sections, can stimulate new ideas and projects for students and researchers seeking to modify or establish research programs.

* To provide teaching/training materials. Both narrative and systematic reviews are effective for teaching graduate students because they cover the research and general aspects of a specific topic in greater depth than one would find in a textbook chapter. Systematic reviews, by definition, are produced using stronger methods and therefore are less prone to bias than narrative reviews. To promote the continued use of research material by their graduate students, educators should set the example by using systematic rather than narrative reviews. Hopefully, then, they will make decisions using evidence derived from studies using the strongest possible methods (evidence-based LIS).

In summary, many reasons exist for producing and using systematic reviews. Many more systematic reviews are being published across disciplines, and in some areas such as health care, education, and psychology researchers and practitioners rely heavily on them. I will now move to a discussion of the production of systematic reviews, the steps that must be followed, and how to search for and find published systematic reviews.

WHAT ARE THE STRENGTHS AND WEAKNESSES OF A WELL-DONE SYSTEMATIC REVIEW?

As with all research, high-quality systematic review articles must conform to accepted methods of production. In most research methodology classifications, systematic reviews are considered to be observational and retrospective. Therefore, they must conform to standard research methods common to all research projects as well as methods unique to observational studies.

First, systematic reviews must be preplanned. This involves development of a protocol that is based on a concise research question and lists the steps in production. The steps must be described in sufficient detail so that those involved in the process understand the tasks and the tasks are completed consistently, correctly, and efficiently. (The steps involved in doing a systematic review are discussed below.) A description of the question and the steps taken must be evident in the published report of the review process so that any reader can identify that careful preplanning was done and replicate the steps if necessary.

Bias is important in observational studies. It can be thought of as any factor, situation, or influence that, when acting alone or together, systematically distorts how we see or report data. Biases take us unknowingly away from the "truth" in research. To overcome or reduce bias, researchers use the strongest methodologies possible (for example, randomized controlled trials). Bias is also reduced by very careful execution of all aspects of the study. To counteract the potential for bias in systematic reviews, researchers who conduct them must emphasize the care they took to develop and carry out the entire process. In addition, they must report the process in detail in their published reports so that any external person can review what they did and even replicate the process to check outcomes.

After setting the question, the steps involved in a systematic review include the identification of potential studies or data sources, selection of studies/sources, data extraction, combining and analyzing the data, and presentation of the findings. Each of these steps is expanded in the next section.

STEPS IN THE PRODUCTION OF A SYSTEMATIC REVIEW

Systematic reviews take considerable time and resources to complete. Broad topics, such as Ondrusek's (2004) review of the research that evaluated end-user online searching behavior with its analysis and synthesis of 175 articles, would take about a year to complete. It is noteworthy that many graduate schools offer courses on systematic reviews and meta-analysis and estimate that it would take approximately 600 hours to complete a narrowly focused review using a team of two to five reviewers. Because of the time needed and skills involved (information retrieval, content expertise, and research methods experience), an interdisciplinary team often works together to produce systematic reviews. Any systematic review project starts with formulation of the question to be addressed.

Question Formulation

All good research is question driven. A well-formulated question for a LIS topic would likely include a description of who was involved (for example, library users, undergraduates, other libraries or librarians), what was being studied (for example, bibliotherapy for bullying, mother-toddler story programs, online instruction for health literacy), the outcomes in which one is interested (for example, increased use of the collection, higher computer literacy), and what studies or data to collect and combine (for example, surveys done by public libraries in cities of similar size to yours, evaluations of online training versus tutorials to increase use of your catalog, randomized controlled trials of giving books to young mothers at well-baby visits). An example of a comprehensive statement or aim from a systematic review relevant to health sciences librarianship is "to establish an evidence base for CL [clinical librarian] programmes ... to determine, from the literature, whether CL programmes

1. are used by clinicians

2. have an effect on patient care

3. have an impact of clinicians' use of literature in practice

4. are cost-effective" (Winning & Beverley, 2003, p. 11).

All members of the team should work to develop and perfect the question because it guides the rest of the review production process. Formulation of the ideal question can take time. Once the question is complete, the review process moves on to identifying potential studies or data sources.

Searching for Studies (Information Retrieval)

The searching or retrieval step is where librarians who are members of or consultants to a systematic review team play a major role. The research question will guide the search process by providing content terms to be translated into structured vocabulary, synonyms, and text words. In conjunction with team members, librarians will have to decide on the most relevant databases to search; years to be included; and limits based on methodology (for example, only randomized controlled trials), geography, language, and patient characteristics (for example, only adolescents). Comprehensive searching also can include hand searching of specific journal titles using the predefined criteria.

Searching is often done in two phases. In the first phase, the goal of the search is to identify published narrative and systematic reviews. If a relevant systematic review is already available, the project could end. If the identified review is on target but older, the research team can build upon the older review and choose not to include studies from it in the newer one, that is, produce an update rather than a complete review. If the reviews retrieved are not exactly on target, they can, at least, provide insight into search terms and database selection as well as potential citations for inclusion in the new review.

After searching for published reviews, the searching proceeds to identify potential original studies. These studies come from three main sources: primary searches in established databases and hand searches of specific journal titles; personal knowledge (team members' reprint files) and personal contact with peers and experts in the field; and "snowballing," whereby the team members find potential citations in bibliographies of reviews and original studies as well as perform citation tracking of important and older studies using resources such as Science Citation Index, Social Science Citation Index, and Arts and Humanities Citation Index. The database and hand-searching procedures are set before the study starts (preplanned) and the "snowball" accumulation occurs as the study progresses. Greenhalgh and Peacock (2005) showed that in 495 studies and systematic reviews of complex health care evidence, 30 percent of the data sources and articles were identified using protocol-based searching methods, 24 percent came from personal knowledge or peers, and 51 percent came from snowballing. Searching done for systematic reviews must be comprehensive and is often complex and iterative.

Comprehensive searching is the foundation of systematic reviews, and librarians are considered to be the experts in this area. I summarize the databases and searching performed by Winning and Beverley (2003) in their review of clinical librarianship. They used nine search phrases in a free-text and thesaurus approach with multiple truncations. They searched many databases in the following areas:

* Medicine (for example, PUBMED, EMBASE)

* Other health care disciplines (for example, Cumulated Index to Nursing and Allied Health Literature [CINAHL], British Nursing Index, Allied and Complementary Medicine Database [AMED], HealthSTAR)

* Science (for example, Science Citation Index)

* Social Science (for example, Social Science Citation Index, Applied Social Science Index and Abstracts [ASSIA])

* Information Science (for example, Library and Information Science Abstracts [LISA], Information Service for Physics, Electronics, and Computing [INSPEC])

* "Grey literature" (unpublished studies and sources) (for example, Health Management Information Consortium, Index to Theses, the National Research Register, Current Research in Britain, and COPAC--catalogs of twenty-four major UK universities plus the British Library, the National Library of Scotland, and the National Library of Wales)

Winning and Beverley (2003) also did citation tracking of identified studies, as well as checking bibliographies of studies and published review articles. Hand searching was done in the Bulletin of the Medical Library Association and Health Information and Libraries Journal. They contacted experts in the field to ask for other published and unpublished studies, an information-retrieval step often included in systematic reviews.

A high-quality systematic review includes a list of each database searched with all limits described, terms used, and other searching processes. For some reviews all of this information is in the published report, while for other reviews a link to a Web site or an invitation for email requests are included.

After the predefined searching is finished, citations are downloaded, combined into one list with duplicates removed, and sorted for easy screening by members of the team. The members often use titles, abstracts, and subject headings to do this initial screening. It is not unusual to have collections of several thousand citations for review using predefined inclusion and exclusion criteria.

Inclusion and Exclusion Criteria to Select Studies for Analysis

The study protocol developed before the searching started needs to include the selection criteria (inclusion and exclusion) that define which articles are to be included in the analyses. By predefining and adhering to the selection criteria, bias in choosing studies for inclusion is minimized. Furthermore, by publishing the criteria along with the search strategies and process, readers can verify that studies were chosen using methods that minimize bias and determine why other studies were excluded. A useful book chapter on systematic reviews (Egger, Dickersin, & Davey Smith, 2001) provides insight into decision making related to selecting studies, as well as a good discussion on publication bias, that is, the propensity of researchers and editors to publish studies that have "positive" results. Trials of "negative" results (those that show no benefit or results that are "disappointing" to the researcher) are published less often. If they are published, a considerable time lag can occur between when the study was done and when it can be found in print. Oftentimes, the journal is of lesser quality and may not even be indexed in the major databases of that discipline. This publication bias leads to inclusion of a higher proportion of studies with positive results while ignoring trials with negative or disappointing findings.

Weightman and Williamson (2005) used the following inclusion and exclusion criteria in their systematic review of the value and impact of information provided through library services for patient care.

Inclusion Criteria 

* Reports that included a formal evaluation using any research methodology

* Services studied were from professionally led libraries

* Services were provided to health professionals

* At least one outcome had to be provided that related to

* Health benefits for patients, members of the public, or both

* Time saved by the health professionals

Exclusion Criteria

* Library services were based only on virtual provision of established resources

* Studies of information skills training

* Specialist training to specific groups of health professionals (e.g., family physicians) outside the traditional library setting (p. 12)

Screening was done on 320 papers, and 28 were included in the published review. Both authors screened studies for inclusion; one author did the initial screening and the second author verified the results and resolved problems. Data checking and duplication of study selection and data extraction are effective methods of minimizing bias. After identifying the studies to be included in the review, each article is obtained in full text. The next step is to extract the data.

Data Extraction

Data extraction from each study or paper is the next step in the process. Using the protocol and its predefined data elements, the team develops a data extraction form to be used by the readers. Each paper is carefully read, often by two people with resolution of differences made through consensus or by bringing in a third party. Data forms in paper and increasingly in electronic format are used to ensure conformity and reproducibility. Some of these forms have multiple pages. Authors of systematic reviews may offer to provide copies of their data extraction forms to anyone who is interested in them. For anyone who is new to the systematic review process, collection of several of these extraction forms from reviews similar to the one you are working on can provide insights and templates.

At the same time as data extraction, individual studies or papers can be evaluated for the quality of their methods. Bias can occur in that lower-quality studies often inflate or exaggerate their findings; thus, analysis of all studies may provide a different answer than analysis of only the high-quality studies. A well-done systematic review will provide data on the method of quality assessments of each study or data source as well as a description of the individual studies and their characteristics. (This quality evaluation may be more important in systematic reviews that are also meta-analyses.)

Data Analysis and Presentation

Data analysis proceeds after data collection. If the data across studies/ papers/data sources can logically (and statistically) be analyzed to provide one final answer to the question, often in numerical form, the systematic review becomes a meta-analysis. Most of the reviews I discuss in this article are non-meta-analysis systematic reviews--the data on the studies are not numerically combined but presented more in a "vote-counting" manner. For example, Weightman and Williamson (2005) extracted the data and summarized what they were across the studies; no numerical combining took place. They stated: "The higher quality traditional library studies ... suggest effects of impacts between 37 and 97% on general patient care, 10-31% on diagnosis, 20-51% on choice of tests, 27-45% on choice of therapy, and 10-19% on reduced length of stay" (p. 4).

These data on improved care and patient outcomes from studies show the worth of health sciences librarianship and are impressive. Systematic reviews such as this one set the standard for other branches of librarianship to provide evidence of their worth using systematic review techniques. Not everyone has the resources to be able to complete a high-quality systematic review. Many of them, however, have been done and can easily be found, especially by librarians with good search skills.

HOW DO I FIND SYSTEMATIC REVIEWS?

Most of the electronic databases include systematic reviews. The examples I have used in this article came from NLM's PUBMED and LISA rather than personal files. In LISA I used variations on the terms systematic review(s), systematic overview(s), and meta-analys(e/i)s as well as metanalysis and metaanalysis. For the health-related databases, several hedges (predetermined search strategies) exist. For example, the University of York in the United Kingdom maintains a database of these hedges for systematic reviews and meta-analyses (http://www.york.ac.uk/inst/crd/search .htm). In addition, the Cochrane Collaboration, a volunteer organization of health professionals (http://www.cochrane.org/index.htm), collects randomized controlled trials in all areas of health care and uses them to publish systematic reviews. Librarians have been involved with the Cochrane Collaboration since its inception in the mid-1990s. The Cochrane Library has over 1,000 systematic reviews, including several of interest to librarians. The library also includes resources to help those who want to learn more about systematic reviews and their production. The Campbell Collaboration (http://www.campbellcollaboration.org/index.html) is a similar volunteer organization whose mandate includes collection of studies and production of systematic reviews in areas of education and social and behavioral sciences. Some of their systematic reviews are relevant to librarians (for example, impacts of after-school programs on student outcomes). The Campbell Collaboration provides opportunities to learn about systematic reviews and to publish library-related reviews. The Database of Abstracts of Reviews of Effects (DARE; http://www.york.ac.uk/inst/crd/crddatabases .htm) provides abstracts from systematic reviews as well as training material for those interested in systematic reviews and meta-analyses in health care, which is defined very broadly. DARE is produced by' the Centre for Reviews and Dissemination at the University of York. Librarians have been involved with DARE since its inception. Access is free and efficient; it is a good place to look for published systematic reviews.

WHAT SORT OF MATERIAL CAN GO INTO SYSTEMATIC REVIEWS?

Many systematic reviews exist, and the studies or data sources that can be effectively integrated to produce new knowledge are almost limitless. The majority of the systematic reviews I have discussed have selected and analyzed quantitative studies. Qualitative studies can also be synthesized using the techniques listed above. An excellent example of a qualitative systematic review summarizes parental attitudes toward childhood vaccination (Mills, Jadad, Ross, & Wilson, 2005). Worries about adverse effects and pain are major considerations for parents. With this review, those who provide immunization can better meet parental information needs and develop effective marketing methods.

A number of examples related to health care show the variety of material that can be integrated, such as an analysis of twenty published definitions of drowning (Papa, Hoefle, & Idris, 2005), and portraits from the fourteenth to the twentieth century to assess disease frequencies (Als et al., 2002). Systematic reviews can also be integrated into other systematic reviews when an abundance of information is present. This type of review is called a "meta-meta-analyses," or a systematic review of systematic reviews (Katerndahl & Lawler, 1999).

RESOLUTION OF THE SCENARIO

At the start of this article we looked at a scenario where you, as the new director of a branch library, had received two important requests that could be addressed by either the production of new systematic reviews or identification of existing ones. You did some searching across health and psychology databases and found at least six high-quality systematic reviews that address the question of the effectiveness of bibliotherapy programs for a wide variety of conditions. The library director was very impressed with your searching skills and greatly appreciated these reviews that reduced his/her workload.

You also found an incredibly detailed systematic review that outlines evidence (randomized controlled trials) of interventions to reduce unintended teen pregnancies (DiCenso, Guyatt, Wilan, & Griffith, 2002). The Public Health director was delighted with the review, in spite of the fact that the evaluated interventions (26 trials in 22 studies) did not show reductions in the rate of intercourse or pregnancy or improvement in the use of contraception. The Public Health director decided that the review was sufficient and no further evidence needed to be gathered or produced. You then decide that you will sign up for a systematic review course at your local university that fall, knowing that knowledge and experience with systematic reviews would be good for your career and job.

SUMMARY

Systematic reviews are an important research method for librarians. These reviews are designed to collect evidence on a given topic from multiple sources using recognized and strong methods to minimize bias. By combining data and information from the collection of varied sources, established information is summarized and integrated and new information is obtained. By applying standard methods of research to avoid or minimize bias in data collection and analysis, we can advance knowledge and improve our services and programs. Librarians need to use and understand systematic reviews both inside librarianship and as partners in the production of high-quality systematic reviews in other disciplines. We already have many of the skills needed, and with some practice and training, we can become effective producers and users of systematic reviews.

REFERENCES

Als, C., Stussi, Y., Boschung, U., Trohier U., & Waber, J. H. (2002). Visible signs of illness from the 14th to 20th century: Systematic review of portraits. BMJ, 325(7378), 1499.

Anderson, L., Lewis, G., Araya, R., Elgie, R., Harrison, G., Proudfoot, J., et al. (2005). Self-help books tot depression: How can practitioners and patients make the right choice? British Journal of General Practice, 55(54), 387-392.

Cook, D. J., Mulrow, C. D., & Haynes, R. B. (1997). Systematic reviews: Synthesis of best evidence for clinical decision. AnnaLs of Internal Medicine, 126(5), 376-380.

DiCenso, A., Guyatt, G., Wilan, A., & Griffith, L. (2002). Interventions to reduce unintended pregnancies among adolescents: Systematic review of randomised controlled trials. BMJ, 324, 1426-1434.

Douglas, R. M., Hemila, H., Chalker, E., & D'Souza, R. R. D., & Tracy, B. (2006). Vitamin C for preventing and treating the common cold. Cochrane Database of Systematic Reviews, 4, CD000980.

Egger, M., Dickersin, K., & Davey Smith, G. (2001). Problems and limitations in conducting systematic reviews. In M. Egger, G. Davey Smith, & D. G. Ahman (Eds.), Systematic reviews in health care: Meta-analysis in context (2nd ed., pp. 43-68). London: BMJ Books.

Elliott, P. (1999). Research into public libraries and ethnic minority communities in the UK, from Clough and Quarmby tn Roach and Morrison: A review article. Journal of Librarianship & Information Sciences, 31(3), 174-177.

Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(1), 3-8.

Greenhalgh, T., & Peacock, R. (2005). Effectiveness and efficacy of search methods in systematic reviews of complex evidence: Evidence audit of primary sources. BMJ, 331(7524), 1064-1065.

Gregory, K. E., & Vessey, J. A. (2004). Bibliotherapy: A strategy to help students with bullying. Journal of School Nursing, 20(3), 127-133.

Harris, M. (2005). The librarian's roles in the systematic review process: A case study. Journal of the Medicol Library Association, 93(1), 81-87.

Katerndahl, D. A., & Lawler, W. R. (1999). Variability in meta-analytic results concerning the value of cholesterol reduction in coronary heart disease: A meta-meta-analysis. American Journal of Epidemiology, 149, 429-441.

Kesling, K. L., & Reinker, K. A. (1997). Scoliosis in twins: A meta-analysis of the literature and a report of six cases. Spine, 22(17), 2009-2014.

Mills, E., Jadad, A. R., Ross, C., & Wilson, K. (2005). Systematic review of qualitative studies exploring parental beliefs and attitudes toward childhood vaccination identifies common barriers to vaccination. Journal of Clinical Epidemiology, 58, 1081-1088.

Moynihan, R. (2004). Evaluating health services: A reporter covers the science of research synthesis. Retrieved November 2, 2005, from http://www.milbank.org/reports/2004Moynihan/ Moynihan.pdf.

Ondrusek, A. L. (2004). The attributes of research on end-user online searching behavior: A retrospective review and analysis. Library & Information Research, 26, 221-265.

Papa, L., Hoefle, R., & Idris, A. (2005). Systematic review of definitions of drowning incidents. Resuscitation, 65, 255-264.

Pearson, K. (1904). Report on certain enteric fever inoculation statistics. British Medical Journal, 3, 1243-1246.

Weightman, A. L., & Williamson, J. (2005). The value and impact of information provided through library services for patient care: A systematic review. Health Information & Libraries Journal, 22(1), 4-25.

Winning, M. A., & Beverley, C. A. (2003). Clinical librarianship: A systematic review of the literature. Health Information & Libraries Journal, 20(Suppl.1), 10-21.

Ann McKibbon is Associate Professor in the Department of Clinical Epidemiology and Biostatistics, Faculty of Health Sciences, at McMaster University. She has worked as a science and health sciences librarian in academic and special libraries and as a librarian researcher in a medical school for thirty years. She received her Ph.D. in medical informatics in 2005 by studying how risk and uncertainty in primary care physicians affects use of electronic information resources. She has taught evidencebased health care to librarians and health professionals over the past twenty years. She has also produced information resources for physicians and nurses and has done research into the production of hedges to identify clinically relevant and important studies in medical and other health databases. She is currently interested in how health professionals who use electronic information resources obtain incorrect answers to their questions. She has published two books and fifty peer-reviewed journal articles, which include several systematic reviews.