Printer Friendly

Comparison of three web-scale discovery services for health sciences researching.

Web-scale discovery (WSD) services offer user-friendly search interfaces, relevance ranking, and large, centralized indexes, allowing rapid, simultaneous searching of multiple library resources. These features pose attractive advantages to health sciences library users, including undergraduate and graduate students as well as health sciences faculty. In addition to ample literature demonstrating undergraduates' strong tendency to opt for the convenience and familiarity of the web over library databases [1-5], research has demonstrated that graduate students, including those in the health sciences, prefer Google as a search tool due to its speed and ease of use [6-8], and health sciences faculty are likewise increasingly using Google and Google Scholar for research [9]. Academic researchers frequently choose these tools over traditional bibliographic databases because of their efficiency and relevance ranking [10]. WSD tools, which allow users to search the majority of a library's resources conveniently through a single search box, merit further investigation from health sciences librarians due to users' demonstrated preference for this easy-to-use, relevance-ranked format.

However, to this point, literature on the applicability of WSD tools for health sciences users has been slim. Implementation case reports speak to the time- and labor-intensive process of customizing a WSD product and conducting usability testing, but also confirm the product's potential to better meet library users' needs [11] and deliver a satisfying user experience [12]. Crook investigated the effectiveness of OCLC's WSD tool, WorldCat Local, as a potential research tool for busy clinicians, using simple keyword searches to test known-item searching [13]. The study found that WorldCat Local could successfully be used to locate material by both title and topic, and emphasized the need for further research on relevance ranking. Most recently, Ketterman and Inman conducted side-by-side search comparisons of PubMed and ProQuest's Summon service [14]. The authors did not evaluate the relevance of search results, but concluded through comparison of journal coverage and currency, that Summon is a valid research tool for the health sciences. While Ketterman and Inman did not recommend it as a replacement for PubMed, they argued that Summon was one of many resources that the library should promote to its users, when applicable to their needs.

A logical next step in this nascent area of study is to investigate the effectiveness of WSD search tools in returning relevant results on topics of interest to health sciences researchers. The purpose of this study is to compare multiple WSD tools, evaluating their respective abilities to return scholarly articles related to health sciences search queries and abilities to assess duplicate search results. It will also provide a glimpse into each product's overlap with MEDLINE content. The primary focus of this study is not to compare the search functionality of WSD tools with that of PubMed/MEDLINE, nor will it investigate WSD tools as a potential substitute for PubMed/ MEDLINE (or any other traditional biomedical database). Instead, it is primarily intended to serve as a resource for libraries that serve large populations of health sciences users, presenting a side-by-side comparison of existing discovery products in order to provide quantitative, evaluative information in the most evenhanded manner possible.

METHODS

We evaluated WSD services from three major commercial vendors: Ex Libris's Primo, ProQuest's Summon, and EBSCO Discovery Service (EDS). Each of these three services were evaluated at two separate institutions that had implemented the tool, resulting in six data collection sites. Research sites were chosen based on the following criteria: libraries were selected only from academic institutions with a Carnegie classification of "Research University High" or "Very High," and only universities with health sciences graduate programs, including a college of medicine, were considered. From this relatively small set of universities, we contacted those with WSD tools in the authors' state and the five surrounding states until we received permission to collect data at six libraries: two with EDS, two with Primo, and two with Summon. Our own institution was included. Two implementations were tested for each product to control for different institutions enabling different features. By collecting data from more than one institution, it would be easier to determine which data were outliers and which were generalizable to all implementations of the products.

Relevance of search results

With the aim of determining the relevance of search results retrieved by each WSD tool, we created questions for 6 major health sciences disciplines: applied health sciences, dentistry, medicine, nursing, public health, and pharmacy. We created 3 reference questions per discipline. We ran searches on the 18 health sciences reference questions at the 6 research sites and thus collected data from a total of 108 search queries.

The reference questions originated from (a) real student and faculty questions posed to liaison librarians at the reference desk or over email, (b) recent University of Illinois at Chicago faculty publication topics, or (c) liaison librarian expertise. All searches were completed between January 14 and February 13, 2015.

We translated each reference question into a simple, short keyword search string (Appendix A, online only), representative of typical student search techniques [15-19]. Commonly used medical abbreviations taken from student and faculty reference questions, such as CHF and EKG, were entered as is to test each WSD tool's ability to interpret keywords and search alternate terms, similar to PubMed's Automatic Term Mapping feature. We entered each search query in the WSD tool and captured the first page of results, as studies have shown this is typically where students stop reading [15, 16, 20-22]. Since each search tool displayed a different number of records on the first results page, and Summon's recently upgraded 2.0 platform inserts no page breaks at all, instead displaying all results on a single screen (a.k.a., "infinite scrolling"), we captured the first twenty results to represent the first "page" of results from each search.

We independently rated each captured reference and marked it relevant, irrelevant, duplicate, or non-scholarly; each of these four classifications was mutually exclusive. Any article considered at least possibly relevant, based on title or abstract, was marked as such. Relevance was judged based on the simple criterion of whether the article appeared to address the proposed research question. Full text was accessed when necessary to determine relevance.

Duplicate results were preserved in the data set because their presence on a results screen negatively impacts user experience. Counting them among search results was, therefore, of interest in determining each WSD tool's ability to deduplicate results and retrieve unique, relevant citations in response to a given user query.

Any disagreements in our ratings were discussed and reconciled. We compared and discussed ratings after we had evaluated the first 40 references, then after the first 10% (220 references). Cohen's kappa ([kappa]) was 0.543 and 0.553, respectively, at these 2 points, indicating moderate observer agreement, where [kappa]=1.000 indicates perfect observer agreement and [kappa]=0 indicates actual observer agreement that does not differ from that which would occur by chance [23]. In total, 2,086 citations were evaluated, [kappa]=0.725 (good observer agreement). The noticeable improvement in agreement indicates that our discussion helped to resolve inconsistencies in our respective evaluation methods.

MEDLINE retrieval and coverage

To compare each WSD tool's retrieval and coverage of MEDLINE content, we searched 1 research topic from each of the 6 health sciences disciplines in PubMed, using a combination of keywords and Medical Subject Headings (MeSH) (Appendix B, online only). These search strings favored precision in order to produce a highly relevant set of articles--a "gold standard"--against which to compare the WSD search results. In addition to our own assessment of the 2,086 citations collected during WSD searching, this gold standard set from MEDLINE would serve as an external indicator of relevance for a smaller subset of the WSD results. That is, we used it to judge how many relevant MEDLINE results the WSD tool could retrieve with a simple keyword search.

For each of the six MEDLINE search queries, we cross-checked results with the first page of results from the corresponding WSD query; for example, we compared the search "tamsulosin ureteric stones expulsion" (RXQ1) in the WSD tool to "tamsulosin"[Supplementary Concept] AND "Ureteral Calculi/drug therapy"[MeSH] (PMQ6) in PubMed. We did not do this to compare the search effectiveness of the WSD tools with that of PubMed, as our search of the latter was a more precise search of a narrower index of sources and a direct comparison of the two would not be valid. Instead, the intended goal was to investigate whether one WSD tool was more successful than the others in retrieving relevant MEDLINE material.

To test each product's overall coverage of MEDLINE content in its central index, we captured the first 50 references from each of these 6 results sets (300 total citations) into an Excel spreadsheet. When collecting data at each research site, we searched for these 300 citations as single, known-item searches to determine whether or not they were indexed by the WSD tool. These MEDLINE citations were searched first by article title. Those not located by title were searched by multiple combinations of author names, journal titles, and keywords, until either we retrieved the citation or were confident that all search options had been exhausted and the citation could be marked as "not indexed." These searches were conducted with no other limits placed on the query, so as to access the entirety of the WSD tool's index.

Process

One notable feature of WSD tools that had to be controlled for was each institution's ability to customize its implementation to accommodate its collections and preferences. To control for such customizations, we took the following steps: first, we ran identical search queries on two distinct implementations of the same service, allowing us to compare and contrast results and note any differences that might be attributed to institutional customizations. Second, we determined that the inclusion of local collections and integrated library system (ILS) holdings in results would cause the widest variance in searches among implementations.

By limiting our queries to scholarly and peer-reviewed journal articles, we excluded these records, avoiding the issue of local holdings altogether. In cases where institutions offered their WSD search through both a general or multidisciplinary interface as well as through a custom search interface for medical disciplines (an option available from all three vendors), we selected the general or multidisciplinary profile in order to maintain consistency across research sites.

RESULTS

Only five of the six research sites are reported on here. At the time of data collection, Summon's 2.0 platform had been recently made available, but its original platform was still in place at many institutions that had not yet chosen to upgrade. One of the two Summon implementations tested for this study was running on what is now considered its legacy platform. The pending retirement of this version renders the corresponding data obsolete, and the legacy platform data have, therefore, been excluded from analysis. The second implementation of Summon tested had been upgraded to the newer 2.0 platform, so the corresponding data set remains. A final total of five sites were included for analysis: two EDS, two Primo, and one Summon. All references to "Summon" that follow below refer to its 2.0 platform.

Relevance of search results

All 3 WSD tools at the 5 locations returned between 50.0% and 60.0% relevant results (Table 1). The 2 implementations of EDS returned the highest number of relevant articles on the first results page (i.e., the first 20 retrieved items), followed closely by Summon. Primo returned a noticeably lower number of relevant results compared with the other 2 products, largely impacted by the frequency of duplicates among Primo results. While EDS and Summon demonstrated similar success rates in deduplication (between 96.0% and 97.0% unique citations on the first page of search results), both instances of Primo returned considerably fewer unique results: 84.9% and 82.9%, respectively. The remaining results were reappearances of previously displayed citations.

MEDLINE retrieval and coverage

All three WSD tools retrieved relevant literature that was not indexed in MEDLINE, as well as relevant literature that was indexed in MEDLINE but not retrieved by our precision searches (Table 2). This could have been caused by several factors, among them the use of different keywords between PubMed and the WSD tools or the use of MeSH in precision searches eliminating recent literature that was not yet indexed. EDS and Summon retrieved a larger number of relevant citations from non-MEDLINE source titles than did either implementation of Primo. One EDS implementation's keyword search results had the most results in common with our precision searches in MEDLINE (31), followed by Summon and the second EDS implementation (25).

The largest overlap between the WSD tools' central indexes and the set of 300 MEDLINE citations was observed with EDS (Table 3). We successfully retrieved all 300 citations in one EDS implementation, while in the second implementation of EDS, we retrieved 299 of the 300 citations. Summon and the Primo services returned a lower percentage of the references, neglecting to index between 8.0% (Summon) and 23.3% (PRIMO2) of the MEDLINE citations sought. The disparity in results between Primo implementations may be attributable to decisions made to activate or deactivate individual publications in the Primo Central index, based on institutional subscriptions.

Total number of search results

The total number of search results was noted for each search query. Primo consistently returned the fewest search results, with a median results set of 59 references in both implementations and an average of 1,668 (PRIMO1) and 1,379 (PRIMO2). Both Summon and EDS returned notably higher numbers: median result totals ranged from 1,356 (Summon) to 3,997 (EDS1), and averages ranged from 2,945 (Summon) to 9,141 (EDS1). In the case of EDS, deduplication occurs as the user scrolls through results, so the final number of unique results is fewer than the initial figure noted. However, it is safe to remark that, regardless of this feature, EDS consistently returned higher numbers of unique references than did Primo. Its initial results numbers, and possibly total unique results, were also higher than those of Summon.

Mapping of less frequent terms

The search "mandatory EKG preparticipation"--taken from a reference question on administering EKGs to athletes before participation in sporting events--returned the fewest results of any query across all 6 tools. In EDS and Primo, the WSD tool searched the abbreviation EKG without substituting alternatives. The low number of results can presumably be attributed to the infrequent use of the term "EKG" in the literature, as compared with "ECG," "electrocardiogram," or "electrocardiography." When this query was run in Summon 2.0, the platform's Automated Query Expansion feature mapped the abbreviation to "electrocardiography," automatically searching this term in addition to "EKG." This produced 97 results, compared with 12 total results in the search on Summon's legacy platform, which did not employ the term mapping feature. In addition, this was the only query out of 108 total searches that returned an entirely relevant "first page" of results, that is, all 20 of the first 20 results were unique, relevant scholarly journal articles.

DISCUSSION

In all areas of evaluation, EDS results appear slightly superior to those of Primo and Summon. EDS returned the highest overall number of relevant results; indexed the highest total number of MEDLINE citations; and in six search queries, EDS both found the most relevant results from MEDLINE source titles and tied with Summon for the most relevant citations found in less-common, non-MEDLINE-indexed sources. However, these results reflect a small sample and were not exceptional to a degree that would definitively recommend EDS over the other two WSD products.

Comparison of results from isolated implementations of each product (i.e., EDS1 versus EDS2, PRIMO1 versus PRIMO2) reflect similar search results, despite the fact that librarians at every research site confirmed having customized their implementation of the product. This suggests that results retrieved from the selected research sites would not likely differ from other product implementations. Numbers differed slightly between Primo MEDLINE indexing rates, as noted above. This divergence aside, the authors did not notice any institutional customizations that they believe to have significantly impacted the results of the study.

We had anticipated that each library's decision to include or exclude certain collections from the searchable index might substantially impact search results. However, these customizations either did not have a noticeable impact on search results or the same changes had been made at both research sites for each product, making institutional modifications of the WSD search index indiscernible from the "out-of-the-box" or standard version in any of the implementations examined. Given that searches were run at only two libraries per WSD tool, it is not possible to know with certainty that all implementations would produce similar results.

While the WSD products varied widely in the total number of search results retrieved, this detail is irrelevant to the majority of searchers, who do not look past the first or second page of results [15, 16, 20-22]. It is, therefore, more pertinent to consider the relevance and quality of the first page of search results when evaluating these products. In these aspects, a few notable observations emerge.

Relevance was observed multiple times to be a function of the topic itself, rather than of the WSD tool. For example, queries with extremely specific terms, such as drug names, were across the board more successful than those using more ambiguous terms. A topic's coverage in the literature also seemed to have a direct effect on the relevance of search results. When a topic was not as widely covered, first-page citations resulted from the WSD tool searching beyond abstract and metadata for search terms. On several occasions, an irrelevant article was retrieved because one of the search terms simply appeared in the full text, generating a "false positive" result.

The success of the search in Summon 2.0 using its Automated Query Expansion feature indicates a promising direction for all WSD vendors to investigate. Maximizing the controlled vocabularies of the databases that contribute content to WSD tools' indexes could help "interpret" simple keyword searches, in the same way that PubMed uses Automatic Term Mapping to enhance user queries. At the time the searches were conducted, Summon was the only product of the three tested offering this feature, although EDS has since added controlled vocabulary enhancements to its search features.

As noted, Primo returned a significantly higher instance of duplicate citations, which negatively impacted the number of unique relevant citations counted for that tool. This deviation notwithstanding, none of the three WSD tools demonstrated overwhelmingly better or worse success than the others in returning relevant results. It would be helpful to have a benchmark to which the relevance numbers could be compared in order to place the success rate of 50%-60% relevant results in context. Unfortunately, due to fundamental differences between WSD search tools and traditional databases, it was not possible to use existing research on the relevance of traditional database search results to compare the two. A product designed to be searched differently by users must necessarily be evaluated differently. For example, in the process of designing the study, we observed multiple times that a short keyword search that delivers multiple relevant results in a WSD search might not retrieve a single record in PubMed. To judge one resource within the context of the other would be unfair.

In the same vein, relevance numbers would almost certainly have been higher if the study design had incorporated the use of product features such as Boolean operators, search limiters and facets, and field searching, all of which were available in all three WSD tools. However, to mimic typical student searching in the present study, we deliberately avoided advanced search constructions requiring a high level of skill and often intensive prior instruction, and is, therefore, irrelevant to the question at hand. The appropriate question to pose when evaluating a resource is not how well it functions under ideal conditions when operated by an expert user, but rather what can be expected when operated in the most likely of circumstances. Under this examining lens, achieving an average of 50%-60% relevance among the first 20 results of every search implies a strong chance that the typical, inexpert user would be able to find 1 or more satisfactory sources on the first page of results. Note that this is accomplished with a simple search of 3 to 6 keywords. While librarians always hope for the opportunity to provide instruction on advanced searching, the data resulting from this study indicate that students and faculty could successfully use WSD tools to start or supplement the research process, in the event that searching instruction were either not available or not desired.

Perhaps the greatest evidence of WSD's value to health sciences researchers was demonstrated when all three tools unexpectedly retrieved, by simple keyword search, a considerable amount of relevant MEDLINE literature that had not appeared in our precision searches in PubMed. In this way, it could be argued that WSD tools may be more effective than PubMed/MEDLINE or other traditional databases in certain cases: chiefly for inexpert users, but also as a supplementary source for those conducting comprehensive literature reviews. The ease with which all three products uncovered literature that was missed with a highly skilled search of MEDLINE, as well as relevant literature from less-common sources that the average user might not think to search (or know how to access), illustrates the appeal of this search format and encourages the recommendation of WSD tools to library patrons. Results suggest that health sciences students with information needs ranging from simple to complex will be able to locate relevant content--from both MEDLINE and less-common sources--quickly, with a simple keyword search from the library home page.

It would have been ideal to include OCLC's WSD tool, WorldCat Local, in this study; however, it was excluded for two reasons. First, no libraries fitting the profile for inclusion in the study were subscribers to the product at the time the study took place, making real-world implementations unavailable for testing. Second, in light of the fact that future customers will adopt the new WorldCat Discovery service, as WorldCat Local is to be retired [24], we decided not to pursue evaluation of a product that would not be available to new customers. Instead, we hope to update the current study in the near future, after WorldCat Discovery has been implemented at major libraries in our region.

Limitations

The study's methodology was limited in that we evaluated only 18 search queries for relevance and searched only 300 MEDLINE citations in each WSD tool's index. The preference for a larger data set was balanced against the amount of data that we felt we could collect at a research site within a limited amount of time and analyze within a reasonable timeframe. Additionally, search queries were constructed by librarians, not taken directly from actual search query logs, though every effort was made to imitate real-world student and faculty search technique and syntax, as we have observed in the literature and in practice.

It should be noted that we are not subject experts in the respective disciplines of each query that we evaluated. However, the high inter-rater agreement measured after all 2,086 citations had been evaluated indicates that our inclusion and exclusion criteria, along with reference to full text when necessary, are a reliable gauge of relevance.

While results from precision searches of MEDLINE are referred to here as a "gold standard" set of articles, they are but one example of a highly specific topical MEDLINE search. These searches were constructed by one person and were not vetted by any external authority. Only six topics were searched in MEDLINE; these data therefore may not be generalizable to other scenarios. Because the primary goal in constructing these searches was that of precision--to produce mostly, if not exclusively, relevant search results--the WSD tools' ability to retrieve MEDLINE literature not found in a "comparable" PubMed search may be overrepresented. A more valid comparison would be to construct PubMed search strings that are better balanced in precision and recall; comparing these results to those of keyword searches in WSD tools might reflect different numbers and would make an interesting subject for future research. Similarly, while excluding local holdings from WSD search results allowed for generalizability, including them may retrieve more uncommon (non-MEDLINE) sources in response to the average search query.

Future directions

For libraries considering adopting a WSD service and for early adopters now contemplating a change in products, there are points to consider beyond the quantitative factors measured here. Chief among these is usability, which has a substantial impact on the desirability of any resource. We observed several usability issues during the collection of data for our study, including a wide variance in compatibility with citation management tools, problems with "freezing" and error messages, and basic differences in interface and user options. These would have an obvious influence on the decision-making process when selecting a WSD tool, but we did not describe them here because they were not the aim of our study. In addition to usability, other factors such as compatibility with a library's existing electronic resource management tools [25], cost, or manpower required for setup and ongoing maintenance may also ultimately influence this choice.

DOI: http://dx.doi.org/ 10.3163/1536-5050.104.2.004
APPENDIX A

Web-scale discovery search queries

Question                                         Keyword 1

AHQ1. In older adults, is there a relationship   older adults
  between anaerobic exercise/strength
  training and accidental fall prevention?
AHQ2. What protective measures should be         electronic health
  taken into account in order to ensure that     records
  Health Insurance Portability and
  Accountability Act (HIPAA)-protected
  information is properly secured in
  electronic health record (EHR) systems?
AHQ3. What is the effectiveness of task-         task-specific
  specific training for patients with upper-     training
  extremity paresis?
DNQ1. How does bleaching affect the              bleaching
  presence of dental caries?
DNQ2. Is a manual or electric toothbrush         manual
  more effective for elderly patients?
DNQ3. When is prophylactic extraction of the     prophylactic
  third molar recommended?                       extraction
MDQ1. Should EKGs be adopted as a                mandatory
  mandatory component of preparticipation
  screenings for athletes?

MDQ2. What is an effective method of             elderly
  screening elderly patients for suicidal
  ideation?
MDQ3. What is the recommended treatment          tuberculosis
  for tuberculosis in HIV-positive patients?
NRQ1. What are the most effective                nursing
  management techniques for nurse                management
  administrators who supervise a diverse,
  multigenerational staff?
NRQ2. Does patient education have a direct       patient education
  effect on the instance of readmission of
  congestive heart failure (CHF) patients?
NRQ3. What is the relationship between           patient positioning
  patient positioning and the presence of
  bedsores (pressure ulcers)?
PHQ1. Is there a significant disparity in        breast cancer
  breast cancer mortality rates between
  African-American and White populations?
PHQ2. What is the efficacy of electronic         electronic
  cigarettes as a smoking cessation aid?         cigarettes
PHQ3. What does the research indicate            soft drink
  regarding the correlation between
  soda/soft drink taxes and weight
  outcomes?
RXQ1. Can tamsulosin be used for the             tamsulosin
  expulsion of ureteric stones?
RXQ2. What research exists regarding the use     naltrexone
  of naltrexone for the treatment of
  Parkinson's disease?
RXQ3. How do high-potency antipsychotic          high potency
  compounds compare to low-potency
  antipsychotics when used to treat
  schizophrenia?

Question                                         Keyword 2

AHQ1. In older adults, is there a relationship   strength training
  between anaerobic exercise/strength
  training and accidental fall prevention?
AHQ2. What protective measures should be         HIPAA
  taken into account in order to ensure that
  Health Insurance Portability and
  Accountability Act (HIPAA)-protected
  information is properly secured in
  electronic health record (EHR) systems?
AHQ3. What is the effectiveness of task-         upper-extremity
  specific training for patients with upper-     paresis
  extremity paresis?
DNQ1. How does bleaching affect the              dental caries
  presence of dental caries?
DNQ2. Is a manual or electric toothbrush         electric
  more effective for elderly patients?
DNQ3. When is prophylactic extraction of the     third molar
  third molar recommended?
MDQ1. Should EKGs be adopted as a                EKG
  mandatory component of preparticipation
  screenings for athletes?

MDQ2. What is an effective method of             suicidal ideation
  screening elderly patients for suicidal
  ideation?
MDQ3. What is the recommended treatment          HIV-positive
  for tuberculosis in HIV-positive patients?
NRQ1. What are the most effective                diverse
  management techniques for nurse
  administrators who supervise a diverse,
  multigenerational staff?
NRQ2. Does patient education have a direct       CHF
  effect on the instance of readmission of
  congestive heart failure (CHF) patients?
NRQ3. What is the relationship between           pressure ulcers
  patient positioning and the presence of
  bedsores (pressure ulcers)?
PHQ1. Is there a significant disparity in        mortality
  breast cancer mortality rates between
  African-American and White populations?
PHQ2. What is the efficacy of electronic         smoking cessation
  cigarettes as a smoking cessation aid?
PHQ3. What does the research indicate            taxes
  regarding the correlation between
  soda/soft drink taxes and weight
  outcomes?
RXQ1. Can tamsulosin be used for the             ureteric stones
  expulsion of ureteric stones?
RXQ2. What research exists regarding the use     Parkinson's
  of naltrexone for the treatment of             disease
  Parkinson's disease?
RXQ3. How do high-potency antipsychotic          low potency
  compounds compare to low-potency
  antipsychotics when used to treat
  schizophrenia?

Question                                         Keyword 3

AHQ1. In older adults, is there a relationship   fall prevention
  between anaerobic exercise/strength
  training and accidental fall prevention?
AHQ2. What protective measures should be         information
  taken into account in order to ensure that     security
  Health Insurance Portability and
  Accountability Act (HIPAA)-protected
  information is properly secured in
  electronic health record (EHR) systems?
AHQ3. What is the effectiveness of task-
  specific training for patients with upper-
  extremity paresis?
DNQ1. How does bleaching affect the
  presence of dental caries?
DNQ2. Is a manual or electric toothbrush         toothbrush
  more effective for elderly patients?
DNQ3. When is prophylactic extraction of the
  third molar recommended?
MDQ1. Should EKGs be adopted as a                preparticipation
  mandatory component of preparticipation
  screenings for athletes?

MDQ2. What is an effective method of             screening
  screening elderly patients for suicidal
  ideation?
MDQ3. What is the recommended treatment          treatment
  for tuberculosis in HIV-positive patients?
NRQ1. What are the most effective                multigenerational
  management techniques for nurse
  administrators who supervise a diverse,
  multigenerational staff?
NRQ2. Does patient education have a direct       readmission
  effect on the instance of readmission of
  congestive heart failure (CHF) patients?
NRQ3. What is the relationship between
  patient positioning and the presence of
  bedsores (pressure ulcers)?
PHQ1. Is there a significant disparity in        disparity
  breast cancer mortality rates between
  African-American and White populations?
PHQ2. What is the efficacy of electronic
  cigarettes as a smoking cessation aid?
PHQ3. What does the research indicate            weight
  regarding the correlation between
  soda/soft drink taxes and weight
  outcomes?
RXQ1. Can tamsulosin be used for the             expulsion
  expulsion of ureteric stones?
RXQ2. What research exists regarding the use
  of naltrexone for the treatment of
  Parkinson's disease?
RXQ3. How do high-potency antipsychotic          antipsychotics
  compounds compare to low-potency
  antipsychotics when used to treat
  schizophrenia?

Question                                         Keyword 4

AHQ1. In older adults, is there a relationship
  between anaerobic exercise/strength
  training and accidental fall prevention?
AHQ2. What protective measures should be
  taken into account in order to ensure that
  Health Insurance Portability and
  Accountability Act (HIPAA)-protected
  information is properly secured in
  electronic health record (EHR) systems?
AHQ3. What is the effectiveness of task-
  specific training for patients with upper-
  extremity paresis?
DNQ1. How does bleaching affect the
  presence of dental caries?
DNQ2. Is a manual or electric toothbrush         elderly
  more effective for elderly patients?
DNQ3. When is prophylactic extraction of the
  third molar recommended?
MDQ1. Should EKGs be adopted as a
  mandatory component of preparticipation
  screenings for athletes?

MDQ2. What is an effective method of
  screening elderly patients for suicidal
  ideation?
MDQ3. What is the recommended treatment
  for tuberculosis in HIV-positive patients?
NRQ1. What are the most effective                staff
  management techniques for nurse
  administrators who supervise a diverse,
  multigenerational staff?
NRQ2. Does patient education have a direct
  effect on the instance of readmission of
  congestive heart failure (CHF) patients?
NRQ3. What is the relationship between
  patient positioning and the presence of
  bedsores (pressure ulcers)?
PHQ1. Is there a significant disparity in        African-Americans
  breast cancer mortality rates between
  African-American and White populations?
PHQ2. What is the efficacy of electronic
  cigarettes as a smoking cessation aid?
PHQ3. What does the research indicate
  regarding the correlation between
  soda/soft drink taxes and weight
  outcomes?
RXQ1. Can tamsulosin be used for the
  expulsion of ureteric stones?
RXQ2. What research exists regarding the use
  of naltrexone for the treatment of
  Parkinson's disease?
RXQ3. How do high-potency antipsychotic          schizophrenia
  compounds compare to low-potency
  antipsychotics when used to treat
  schizophrenia?


APPENDIX B

PubMed search query translations

PMQ1. Applied Health (AHQ1)

("resistance training"[MeSH] OR "strength training" OR "anaerobic exercise") AND ("accidental falls" OR "fall risk") AND ("aged"[MeSH] OR "elderly" OR "older adults" OR "senior citizens") NOT "middle aged"[MeSH]

PMQ2. Dentistry (DNQ3)

("Molar, Third"[MeSH] OR third molar*[ti] OR "wisdom teeth") AND (prophylactic extraction* OR prophylactic removal*)

PMQ3. Medicine (MDQ1)

(("Electrocardiography"[MeSH] OR EKG OR ECG) AND (athlete OR athletes OR athletic OR sports) AND (preparticipation OR pre-participation) AND (screening OR evaluation) AND ("mandatory" OR "guidelines" OR "best practices"))

PMQ4. Nursing (NRQ3)

("Patient Positioning"[MeSH] AND "Pressure Ulcer"[MeSH])

PMQ5. Public Health (PHQ1)

(("African Continental Ancestry Group"[MeSH] AND "European Continental Ancestry Group"[MeSH]) AND "Breast Neoplasms/mortality"[MeSH]) AND ("Health Status Disparities"[MeSH] OR "racial differences")

PMQ6. Pharmacy (RXQ1)

("Tamsulosin"[supplementary concept] AND "Ureteral Calculi/drug therapy"[MeSH])

ACKNOWLEDGMENTS

The authors thank Sandra De Groote, AHIP, and Deborah D. Blecic, AHIP, for their generous contributions of time and support throughout this project. They also thank the numerous colleagues who offered feedback on the project, as well as those at the research sites who allowed use of their facilities and answered many questions.

REFERENCES

[1.] Griffiths JR, Brophy P. Student searching behavior and the web: use of academic resources and Google. Libr Trends. 2005 Spring; 53(4):539-54.

[2.] Rowlands I, Nicholas D, Williams P, Huntington P, Fieldhouse M, Gunter B, Withey R, Jamali HR, Dobrowolski T, Tenopir C. The Google generation: the information behaviour of the researcher of the future. Aslib Proc. 2008 Jul 6; 60(4):290-310. DOI: http://dx.doi. org/10.1108/00012530810887953.

[3.] Martin J. The information seeking behavior of undergraduate education majors: does library instruction play a role? Evid Based Libr Inf Pract. 2008 Dec 13; 3(4):417.

[4.] He D, Wu D, Yue Z, Fu A, Vo KT. Undergraduate students' interaction with online information resources in their academic tasks. Aslib Proc. 2012; 64(6):615-40. DOI: http://dx.doi.org/10.1108/00012531211281715.

[5.] Colon-Aguirre M, Fleming-May RA. "You just type in what you are looking for": undergraduates' use of library resources vs. Wikipedia. J Acad Libr. 2012 Nov; 38(6):3919. DOI: http://dx.doi.org/10.1016/j.acalib.2012.09.013.

[6.] Judd T, Kennedy G. Expediency-based practice? medical students' reliance on Google and Wikipedia for biomedical inquiries. Br J Educ Tech. 2011 Mar;42(2):35160. DOI: http://dx.doi.org/10.1111/j.1467-8535.2009.01019. x.

[7.] Connaway LS, Dickey TJ, Radford ML. "If it is too inconvenient I'm not going after it": convenience as a critical factor in information-seeking behaviors. Libr Inf Sci Res. 2011 Jul; 33(3):179-90. DOI: http://dx.doi.org/10. 1016/j.lisr.2010.12.002.

[8.] Cothran T. Google Scholar acceptance and use among graduate students: a quantitative study. Libr Inf Sci Res. 2011 Oct; 33(4):293-301. DOI: http://dx.doi.org/10.1016/j. lisr.2011.02.001.

[9.] De Groote SL, Shultz M, Blecic DD. Information-seeking behavior and the use of online resources: a snapshot of current health sciences faculty. J Med Libr Assoc. 2014 Jul;102(3):169-76. DOI: http://dx.doi.org/10. 3163/1536-5050.102.3.006.

[10.] Olle C, Borrego A. A qualitative study of the impact of electronic journals on scholarly information behavior. Libr Inf Sci Res. 2010 Jul; 32(3):221-8. DOI: http://dx.doi. org/10.1016/j.lisr.2010.02.002.

[11.] Thompson JL, Obrig KS, Abate LE. Web-scale discovery in an academic health sciences library: development and implementation of the EBSCO Discovery Service. Med Ref Serv Q. 2013 Feb 8; 32(1):2641. DOI: http://dx.doi.org/10.1080/02763869.2013.749111.

[12.] Pinkas MM, Del Baglivo M, Klein IR, Brown E, Harris R, Gerhart B. Selecting and implementing a discovery tool: the University of Maryland Health Sciences and Human Services Library experience. J Elec Resour Med Libr. 2014 Jan; 11(1):1-12. DOI: http://dx.doi.org/10.1080/ 15424065.2013.876574.

[13.] Crook L. WorldCat Local and the clinician: is discovery improved? Endnotes. 2011 Nov;2(1):1-11.

[14.] Ketterman E, Inman ME. Discovery tool vs. PubMed: a health sciences literature comparison analysis. J Elec Resour Med Libr. 2014 Jul; 11(3):115-23. DOI: http://dx. doi.org/10.1080/15424065.2014.938999.

[15.] Asher AD, Duke LM, Wilson S. Paths of discovery: comparing the search effectiveness of EBSCO Discovery Service, Summon, Google Scholar, and conventional library resources. Coll Res Libr. 2013 Sep; 74(5):464-88. DOI: http://dx.doi.org/10.5860/crl-374.

[16.] Asher AD, Duke LM. Searching for answers: student research behavior at Illinois Wesleyan University. In: Duke LM, Asher AD, eds. College libraries and student culture: what we now know. Chicago, IL: ALA Editions; 2011. p. 71-86.

[17.] Holman L. Millennial students' mental models of search: implications for academic librarians and database developers. J Acad Libr. 2011 Jan; 37(1):19-27. DOI: http:// dx.doi.org/10.1016/j.acalib.2010.10.003.

[18.] Mischo WH, Schlembach MC, Bishoff J, German EM. User search activities within an academic library gateway: implications for web-scale discovery systems. In: Popp MP, Dallis D, eds. Planning and implementing resource discovery tools in academic libraries. Hershey, PA: IGI Global; 2012. p. 153-73.

[19.] Herskovic JR, Tanaka LY, Hersh W, Bernstam EV. A day in the life of PubMed: analysis of a typical day's query log. J Am Med Inform Assoc. 2007 Mar 1; 14(2):212 20. DOI: http://dx.doi.org/10.1197/jamia.M2191.

[20.] Majors R. Comparative user experiences of nextgeneration catalogue interfaces. Libr Trends. 2012 Summer;61(1):186-207. DOI: http://dx.doi.org/10.1353/lib. 2012.0029.

[21.] Bar-Ilan J, Keenoy K, Levene M, Yaari E. Presentation bias is significant in determining user preference for search results--a user study. J Am Soc Inf Sci. 2009 Jan; 60(1):135-49. DOI: http://dx.doi.org/10.1002/asi.20941.

[22.] Jansen BJ, Spink A, Saracevic T. Real life, real users, and real needs: a study and analysis of user queries on the web. Inf Process Manag. 2000 Mar 1; 36(2):207-27. DOI: http://dx.doi.org/10.1016/S0306-4573(99)00056-4.

[23.] Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960; 20(1):37-46.

[24.] WorldCat Discovery migration [Internet]. Dublin, OH: OCLC Online Computer Library Center; 2015 [cited 3 Jun 2015]. <http://www.oclc.org/worldcat-discovery/ migration.en.html..

[25.] Djenno M, Insua G, Gregory GM, Brantley JS. Discovery usability: comparing two discovery systems at one academic library. J Web Libr. 2014 Jul 16; 8(3):263-85. DOI: http://dx.doi.org/10.1080/19322909.2014.933690.

AUTHORS' AFFILIATIONS

Rosie Hanneke, MLS, rhanneke@uic.edu, Assistant Professor and Assistant Information Services Librarian, Library of the Health Sciences-Chicago, University of Illinois at Chicago, 1750 West Polk Street, Chicago, IL 60612; Kelly K. O'Brien, MLIS+, kkobrien90@gmail.com, Assistant Professor and Regional Assistant Librarian, Crawford Library of the Health Sciences-Rockford, University of Illinois at Chicago, 1601 Parkview Avenue, Rockford, IL 61107

Received July 2015; accepted December 2015

Rosie Hanneke, MLS; Kelly K. O'Brien, MLIS

* Research was performed with no external funding.

** This article has been approved for the Medical Library Association's Independent Reading Program <http://www.mlanet. org/page/independent-reading-program>.

*** Supplemental Appendix A and Appendix B are available with the online version of this journal.

([dagger]) Current contact information: 18120 Wildwood Springs Parkway, Spring Lake, MI 49456.
Table 1

Relevance of first page * of results from 18 search queries

               EBSCO Discovery
               Service (EDS) 1   EDS2   PRIMO1   PRIMO2   SUMMON

Relevant/            195         216     167      169      192
  Included
Excluded             165         144     161      156      168
  ([dagger])
Total                360         360     328      325      360

* "First page" defined as first 20 search results or entire results
set when <20.

([dagger]) Total excluded results comprise irrelevant, duplicate, and
non-scholarly results.

Table 2

Of 120 * results, source of relevant items (MEDLINE vs. non-MEDLINE)

                            EDS1   EDS2   PRIMO1   PRIMO2   SUMMON

Total relevant               82     93      67       65       81
MEDLINE                      70     78      64       62       66
Non-MEDLINE                  12     15      3        3        15
Also retrieved in MEDLINE    25     31      23       22       25
  with authors' precision
  search

* PRIMO1 returned 105 results for the 6 evaluated search queries;
PRIMO2, 101 results.

Table 3

Web-scale discovery coverage of 300 MEDLINE citations

              EDS1   EDS2   PRIMO1   PRIMO2   SUMMON

Indexed       299    300     267      230      276
Not indexed    1      0       33       70       24
Total         300    300     300      300      300
COPYRIGHT 2016 Medical Library Association
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2016 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:PAPER
Author:Hanneke, Rosie; O'Brien, Kelly K.
Publication:Journal of the Medical Library Association
Article Type:Report
Date:Apr 1, 2016
Words:6810
Previous Article:Access to human, animal, and environmental journals is still limited for the one health community.
Next Article:Mapping the literature of hospital pharmacy.
Topics:

Terms of use | Privacy policy | Copyright © 2019 Farlex, Inc. | Feedback | For webmasters