Printer Friendly

Reference materials in LIS instruction: a Delphi study.

Introduction and Problem Statement

The delivery of effective reference services depends on thorough knowledge of reference sources. While in the past reference sources could be clearly identified and grouped into categories, the shift to online tools has created an abundance of sources that defy traditional conventions about what reference sources make up the building blocks of a reference librarian's toolkit. Professors teaching courses in LIS programs and training the next generation of reference librarians struggle to decide which of the many reference sources available are the key ones that will best serve the future careers of LIS students.

A number of studies have examined the issues associated with teaching reference in the current environment of digital sources, but they tend to focus on pedagogy and means of delivery (Agosto, Rozaklis, MacDonald, & Abels, 2010; Mon, Abela, Agosto, Japzon, Most, Masnik, & Hamann, 2008). While studies acknowledge the challenges associated with determining the core collection (Adkins & Erdelez, 2006; Shaw & Okada, 2001), no studies to date have actually attempted to identify the core sources that LIS students should be introduced to as part of their education.

This study attempts to fill this gap by directly addressing this question: what are the sources that LIS students should be introduced to as part of a core reference course?

Effective reference education for LIS students focuses on many aspects of the reference process, including evaluation of sources, user behavior, ethics and policies, articulated in the RUSA Guidelines for Behavioral Performance of Reference and Information Service Providers (RUSA, 2004). These processes also received ample attention in the research literature, yet scant research exists on the reference sources themselves. The RUSA Guidelines address the importance of familiarity with reference sources in section 4.2. The Guidelines emphasize the importance of identifying the appropriate sources, but no further discussion as to their nature.

Reference instructors concluded some time ago that spending time on print sources is less feasible in a time when libraries are sending legacy print collections to remote storage. A more recent challenge is the re-emergence of some legacy tools in digitized format.

While reference materials constitute an aspect of reference education, the research literature of recent years, as demonstrated below, does not address this question.

Prior Research

Prior research that informs this study includes research related to reference education in LIS schools and research regarding future trends in reference services.

Education for Reference in LIS Schools

Research on reference education in LIS schools is concerned primarily with teaching strategies. Studies by Adkins & Erdelez (2006), Agosto et al. (2010) and Mon et al. (2008) have demonstrated many challenges and barriers that reference instructors face. These challenges include selecting sources for curriculum subjects, balancing reference sources and reference processes in a single course, and coping with the wide breadth of content covered in reference courses (Agosto et al., 2010). Additionally, instructors must teach newer types of remote reference services, such as digital media, alongside older modes like telephone reference instead of replacing them (Mon et al., 2008). Another challenge reported by prior research is a loss of familiarity with print sources among students, along with a lack of access to print sources in courses that are completely online (Adkins & Erdelez, 2006), but not all researchers found this to be a concern among educators (Agosto et al., 2010).

Since instructors face the challenge of balancing electronic and print content in reference curriculums, Adkins and Erdelez (2006) conducted a survey among 40 reference instructors that revealed that, overall, instructors spend more time (59%) on electronic sources than on print sources, with subject-specific and online courses spending significantly more time on electronic sources (up to 94%) and general reference dividing the time about equally between electronic and print sources.

Covering reference content and instruction in hands-on practice is another balance that instructors strive to achieve in the classroom. The Adkins and Erdelez (2006) survey found that the two most frequently used methods for introducing students to online sources were in-class searches and discussion of online searching principles that would later be applied in the students' assignments (Adkins & Erdelez, 2006). Another method, described by Shaw and Okada (2001), is the collaborative mode used in LIS reference courses at Indiana University in Bloomington. Lectures were combined with discussion sections and guest presentations by reference librarians from varying backgrounds and philosophies each week. This ensured that students were exposed to a range of perspectives. This approach to teaching reference has many advantages, including consistency in topic coverage as well as the integration of theory and skills application (Shaw & Okada, 2001).

Another challenge facing LIS instructors is the heterogeneity in technology skills of students. One study by Agosto et al. (2010) found great variance in technology skills both among students and among their LIS instructors. This barrier is echoed in the results of a 2009 study of new graduate LIS students that explored their perceptions, attitudes and prior information technology and Web 2.0 experience, as well as the demand for these skills in the workplace (Mon & Randeree, 2009). Among the Web 2.0 technologies used by public libraries for internal and external communications were blogs, wikis, RSS feeds, and social networking. The public librarians surveyed by Mon and Randeree felt that LIS students were not always prepared to deal with these technologies. Of the 31 graduate students surveyed, only two (5.9%) had created content for wikis and only six (17.6%) had created content for blogs.

Student characteristics also need to be considered in teaching reference. Agosto et al. (2010) point to challenges resulting from a wide range of professional experience levels among students, the physical dispersion among students that results in a wide range of background experience and disciplinary preparation, and tunnel-vision attitudes toward learning whereby students wish to focus only on content relevant to their career goals. The final barrier identified results from the nature of reference practices and uncertainty regarding the work environments students will face upon graduation (Agosto et al., 2010).

The Current and Future State of Reference

Researchers are also addressing current and future trends in reference services, examining modes of delivery (Mon et al., 2008) as well as service models and reference sources (Ferrari, 2011). Agosto, Abels, Rozaklis, & MacDonald (2009), Mon et al. (2008), and Ferrari (2011) provide insight into how reference education lines up with the delivery of reference services in different library settings, including academic, public, and special libraries. As in reference education, reference sources are also a concern of practicing reference librarians. Washington State Library conducted a survey among public library staff to determine the electronic sources preferred by librarians and discovered that in addition to more content, librarians want online resources that are more service-based, such as language learning and self-help services (Ferrari, 2011). Some researchers consider the future of reference in the digital realm to be tied to the possible demise of ready reference (Agosto et al., 2009).

One study suggests that reference education may not accurately reflect current modes of remote reference delivery. A survey of 100 public libraries in the United States discovered that telephone was the most common form of remote service offered, while an analysis of remote reference education via syllabi revealed an emphasis on digital modes of delivery rather than analog media (Mon et al., 2008). The future of reference services in virtual environments will likely involve the convergence of multiple modes of reference simultaneously, including face-to-face, telephone, chat, and email (Agosto et al., 2009). Other themes concerning the future of reference services include the burgeoning range of information services, reference as a collaborative process, reference in the library 2.0 mode, and the shift from librarian-as-searcher to librarian-as-evaluator (Agosto et al., 2009).

The Delphi Method

The Delphi method--named for the oracle of Delphi, who was able to foresee the future--is a research method used for forecasting events or trends by soliciting the opinions of experts who refine their forecast based on feedback from other experts in an attempt to achieve consensus or convergence of opinions. The Delphi method is identified by Powell and Connaway (2004) as appropriate when attempting to solve issues that are non-factual, and is regarded by Fisher (1978) as most appropriate for developing value and panel analysis.

Most recently developed by Luo and Wildemuth in 2009, the method is versatile enough that it can be used not only in forecasting but also to solicit experts' opinions and assess the relative salience of issues. Delphi studies are considered appropriate for investigating questions that can benefit from subjective input from a group of highly qualified experts (Luo & Wildemuth, 2009).

While variation in application or goals exists, some characteristics need to be maintained in order for a study to be considered a Delphi study. First, Delphi studies rely on anonymity: participating experts are expected to remain anonymous to each other. The interaction with other panel members is through the researcher or facilitator, who issues reports of results to all members. This controlled feedback, provided as statistical summaries of the groups' opinions, is a second characteristic of the Delphi method. The summaries are shared with the group members, who then reconsider and refine their input in an attempt to achieve consensus or at the very least to narrow the range of responses. A third characteristic is the use of several rounds of data collection, usually three or four, in an attempt to reach agreement among the experts. In each round, questionnaires, or other data collection tools, are modified based on responses from the previous round. Finally, when consensus is not reached, or when responses deviate greatly, participants are asked to provide justification for their stand (Luo, 2009).

While the Delphi method is considered reliable, concern has been expressed with regard to five areas: lack of statistical tests, lack of demographic description of participants, selection of experts, lack of explanatory quality, and degree of anonymity (Luo, 2009). These limitations, and an explanation of the way this research addressed them, are addressed in the next section.

The Delphi method is used regularly by LIS, particularly when the area of investigation is novel or highly specialized (Du, 2009; Kochtanek & Hein, 1999; Missing ham, 2011; Westbrook, 1997) or in cases when researchers are attempting to forecast future trends (Bronstein & Aharony, 2009; Feret & Marcinek, 1999, 2005).

Westbrook (1997) used a Delphi method to conduct an exploratory study on the information needs of researchers in Women's Studies, a then newly emerging transdisciplinary field with little prior research. Kochtanek and Hein (1999) conducted a Delphi study to understand broader issues relating to digital libraries, a then relatively new term that was open to many interpretations. More recently, the Delphi method was used to investigate areas that are highly specialized. Missingham (2011) used the Delphi method to identify key challenges for parliamentary libraries that face changing expectations of citizens' engagement with Parliament and their representatives. Du (2009) conducted a Delphi study to determine if librarians agreed with findings by the National Endowment for the Humanities (NEH) that revealed a decline in literary reading, particularly among young readers. Feret and Macinek (1999, 2005) conducted a Delphi study in 1999 to forecast the future of academic libraries in 2005, and in 2005 repeated the study to examine their initial findings and forecast the next phase of academic libraries.

Research Design

The design of this research followed the steps described above (Luo, 2009) and addressed the anonymity of panelists, the interactions facilitated through the researcher, the optimum numbers of rounds and the justifications of panelists of their choices.

I. The Experts

Panelists were selected based on several criteria: first among them was their expertise in the area of reference services and sources. Expertise was determined by a combination of factors that included workplace experience, professional activities and contribution to the LIS environment through publication and service to the community. Panelists included members who have had recent experience in teaching reference or related courses in LIS schools. In addition, the researcher attempted to find experts from different types of libraries, a variety of subject specialties and from different geographies.

Experts were recruited by personal invitation from the researcher. The research process, the time allocated to complete each round, the number of expected rounds, the duration of the entire study and the time commitment required, were all communicated in advance. All experts accepted the offer to participate. Participants signed an informed consent prior to completing the first round.

A summary profile of the panelists is available in Figure 1, and detailed individual profiles appear in Appendix 1.

II. The Rounds

While in theory the Delphi process can be repeated until consensus is reached, researchers have determined that in most cases three rounds are sufficient (Hsu & Sandford, 2007). The study consisted of a pre-test and four Delphi rounds conducted during spring 2011. Panelists were given 10 days to complete each round with 10 days between each of the rounds.

The pre-test was conducted to test the instrument and identify questions that needed to be refined. Using an online survey tool, the following question was presented to a panel of six experts:

   The purpose of this study is to develop a
   list of reference sources that every student
   graduating from an LIS program should
   be familiar with. For that purpose we are
   asking for your help. Which do you think
   are the "must know" reference sources for
   information professionals?

In addition, as part of the pre-test, experts were asked three questions about their professional experience (see Appendix 2) and were given a list of 117 resources to rank. Panelists were also asked to suggest sources they thought were missing from the list and to provide any additional comments.

In each round the experts were presented with a list of reference sources and were asked to answer a single question.

The researcher developed this list of resources using several sources. It featured reference materials used by the researcher, reference materials covered in two popular reference textbooks as well as a textbook for online searching, and panelists' suggestions. The list of sources was designed for LIS students who are taking a core/required general reference course and have no clear idea of where they will be working in the future.

For each source, panelists answered the question, "How important is it that students graduating from Library and Information Science programs be familiar with the following source?" The experts were asked to select one of the following four options for each source: Very important; Somewhat important; Not that important; Not familiar with source.

III. Limitations

The research design took into account the shortcomings and weaknesses previously identified in Delphi studies (Hsu & Sanford, 2007) and addressed them in the planning stages. A limitation that proved particularly challenging to address was one cited by Hsu and Sanford, noting that "subtle pressure to conform with group rating was one of the major drawbacks in the Delphi study" (Hsu & Sanford, 2007, p. 5). This was addressed by providing panelists the space to comment on the final scores and voice their thoughts on the level of agreement reached. Panelists used this to express reservations, and very characteristically, panelists attributed their reservations to the reality of their own work environment and less to shortcomings in the reference source.


The pre-test used six experts, a number that was expanded to twelve for the four Delphi rounds. In the pre-test, consensus was reached for a total of 17 (out of 117) items and only positive consensus was reached (i.e. "very important"). For a full list of the pre-test consensus items see Appendix 3.

The next four rounds of the Delphi test included 12 experts. The first round listed 130 reference sources. The number of items that received 100% consensus was reduced to seven, although a large number of items received majority. In Round 2 approximately 30 sources were eliminated; these included resources that had complete or high level of consensus. Round 2 produced four sources that had consensus of 100% and an additional 16 sources that had a high level of agreement.

For Round 3, another 20 sources with consensus or near-consensus were eliminated (i.e., the survey did not include sources for which all participants were in agreement as to positive relevance, such as Ulrich's Guide to Periodicals, or for which all participants were in agreement as to lack of relevance, such as Grolier Online). In addition, for Round 3 the scale was modified in a way that would urge panelists to make a choice and not stay in the middle ground.

In Round 3 the "not that important" and "not familiar with source" options were eliminated. Panelists were asked to choose between "important" and "not important." Recognizing that this might be difficult, since there are always sources that fall between the two, a comment box was added to each source. After choosing between "important" and "not important," panelists were encouraged to add their comments. For this round, for the total of 83 sources included on the list, panelists reached consensus or high agreement (80% and above) for 18 sources.

In the last round--Round 4--the tabulated results (see Appendix 4) of sources from all three previous rounds were listed and panelists were asked to provide any comments they had. At the end of this round only a few modifications were made to the list provided in Appendix 4.

The four Delphi rounds achieved consensus on 24% of the sources on the list, and high agreement on an additional 30%. In total, high agreement or consensus was reached on 54% of the items, as shown in Figure 2. Thirty-one items received consensus votes from all 12 panelists and are listed in Table 1. An additional 40 items (30%) received high agreement (80-99% agreement) and are listed in Table 2.

Non-numeric Findings

The pre-test and all four Delphi rounds included the option of adding open-ended comments, and most panelists chose to do so. Their comments provide valuable insight into the ways in which information professionals think of reference materials. The main themes to emerge from the comments are described in this section.

There was consensus and high agreement as to the importance of catalogs and their relation to the notion of bibliographic control, although panelists wanted to emphasize that students be aware of the subtleties that dictate access. For example, one panelist wrote:

   Knowledge of WorldCat is essential,
   though it's important that Library students
   know that not all library collections are
   included. My library, for instance, subscribes
   to OCLC and our collection can be
   found when searching WorldCat through
   OCLC Connexion/FirstSearch, but because
   we don't pay the extra fee, our collection
   cannot be found on the public WorldCat.
   ArchiveGrid is good to know about,
   especially if librarians are working with
   researchers or writers--though technically,
   all the information should also be able to
   be found through OCLC.

Another grouping of sources that received consensus and high agreement from participants was general full-text indexes such as ScienceDirect, JSTOR and PsycINFO. Panelists were very budget-conscious in their selection and included open access sources such as ERIC and SSRN. For example, one respondent said:

   Basically concur with all these ratings.
   ScienceDirect, while hideously expensive,
   is perhaps more important in a university
   setting than the score indicates, and I've
   ranked an open access resource (SSRN)
   highly even though it's specialized--working
   at a public college I am always thinking
   about budget cuts.

As for retrospective indexes, while endorsement was low, some panelists expressed a strong minority opinion that objects to the relatively low rankings. For example: "Strongly disagree about Historical Abstracts. It (and America: History and Life) are much more important for research in many fields than this rating indicates."


Panelists felt that while some tools don't get much use in the day-to-day, LIS students should be aware of their existence, either as part of the legacy tools of reference or for the possibility of future use. For example: "NUC is important as a concept (emphasis added).... ArchiveGrid and BLC are similar: important to know of their existence, but not often necessary in real life transactions." Or:

   I'm pretty sure I marked a few of these as
   very important in prior rounds, but only so
   future librarians know that they exist and
   can be used as a 2nd line of crosschecking.
   The fact is that most of this information is
   just as easily found online and the online
   information is, in fact, more up to date.

The Outmodedness of Ready Reference

Ready reference tools were also identified as tools that LIS students may want to be aware of, but panelists predicted they are not likely to use them in their careers. Several of the comments acknowledged that some reference materials were replaced by quick-and-easy Internet searches. This was particularly true for the category of sources known as "ready reference."

Some characteristic comments were: "I do feel like I use ready reference sources far less often than other reference sources," "Fascinating how lukewarm we all were to most of these. Sounds to me like we're really moving out of the era of the Almanac," and "I have a hard time imagining the context that one reaches for a general almanacs these days, but perhaps a school library." Yet this distinction between what LIS students should know and what they will use continues: "I seldom use many of these resources but do think ... [they] are ones every library student should be familiar with."

While many panelists were somewhat ambivalent about ready reference tools, admitting to using them little but not quite willing to dismiss them, one panelist voiced an opinion by saying she was "Very relieved to see the 'no's' here! I am less thrilled with b. and c.--I think those are holdovers from a previous era, but ... again ... I'm probably just a bad-girl maverick. ..."

Yet, the three dictionaries on the list (Oxford English Dictionary, Webster's and Bartlett's Familiar Quotations) all received consensus, and were described as "essential", "important" and "useful". In the same vein, panelists admitted to little use of encyclopedias, yet ranked them highly. One comment seems to have captured the sentiment of all experts: "Honestly, I cannot remember the last time I used any of these to answer a reference question, but I would be loathe to mark them as unimportant for a library science student to know about."

The Bottom Line (open access)

Although none of the panelists indicated the need to cut resources due to budget constraints, they were overall aware of the advantage of open-access sources, not just to the bottom line, but as the way of the future. For example:

   We have a subscription to RefWorks at
   my library. Looking at this now I'd bump
   Zotero a bit higher--again, the open access
   advantage. Also, I'm starting to hear more
   about Mendeley, though I haven't explored
   it yet. This landscape is changing, esp.
   with the incorporation of social networking
   features into citation management. Maybe
   it's just important for students to know that
   there are open (and free) citation management

The place of work and geographic location of the panelists was reflected in their rankings, particularly of local sources, such as the Almanac of the City of New York. One panelist said: "I suppose the high % of NOs for the Almanac of NYC makes sense ... but the newest edition (2nd edition) is a fantastic resource." Other panelists reflected on the importance of local tools. One participant said, "Directories really have to offer some kind of unique subject or geographic arrangement to be at all useful."

Beyond general reference

Panelists were asked to consider these sources through the lens of a required reference class rather than a specialized one. Several of the comments noted that some tools, while important, were beyond the scope of the generalist. Geographic tools received overall lukewarm endorsement from panelists, who acknowledged the usefulness of historic atlases but admitted to using few geographic tools. A typical comment was, "I am not entirely convinced that geographic resources are important for all library students to know or be familiar with."

Yet despite lack of enthusiasm for geographic tools, another category of specialized tools, namely data and statistical sources, received consensus and high agreement from panelists who commented that "[s]tatistical sources are important and students should be aware of them," and that:

   Numbers are essential to papers and
   presentations. Usually it takes a decent
   amount of time to find the number--more
   than the seconds it will take to say the
   number or include a number in a written
   publication. Mastering these tools will let
   one be more efficient.

Discussion and Conclusion

Results from the panelists indicated some very clear recommendations from reference professionals regarding the sources they identify as important for all LIS students. First, all the sources that received consensus were either born digital or had a digital edition, and in the comments all panelists referred to digital rather then to print counterparts. A second trend to emerge is the high number of article aggregates and compilations, e.g. Oxford Online Reference, Project MUSE or Academic Search Premier. While many of the high-ranking items were aggregates and compilations, a third characteristic of the list is the high number of single-title items such as the Oxford English Dictionary and Encyclopedia Britannica.

While these recommendations are not surprising, there were some choices made by panelists that were somewhat unexpected. First, few geographic tools received high levels of agreement and none received consensus. One panelist thought the geographic tools were perhaps too specialized for a general reference course. The lack of endorsement of geographic tools is in contrast to the consensus reached on other highly specialized tools such as or FDsys, which leads one to hypothesize that it is not the narrow focus of geographic tools that kept them off the list and that another explanation is warranted.

On the item level, some of the opinions of the panelists are in contrast to the items' popularity in libraries. This was particularly true for three titles: ScienceDirect (low agreement), Ancestry (low agreement) and Biography in Context (low agreement). One of the panelists commented on the low placement of ScienceDirect, noting that "ScienceDirect, while hideously expensive, is perhaps more important in a university setting than the score indicates." Ancestry and Biography in Context are quite popular with public library users (see Figure 3); this low endorsement is perhaps a result of the small number of public librarians among the panelists.

A second surprise that emerged from the study was in regard to How-to tools. How-to tools such as Career Cruising, LearningExpress Library and Mango Languages received a resounding "NO" all around. Yet it is important to note that there are some indicators that the high level of agreement here does not reflect the reality of public libraries. For example, the New York Public Library lists some How-to and other money-saving tools such as Freegal Music and Mango Languages as their most popular (Figure 3).

This view was confirmed in a recent study conducted by Washington State Library to find out the resources libraries would like access to. Public libraries indicated a need for such tools. One librarian surveyed said: "Sounds like customers are looking for more tools for job searching, language learning, and downloadable audio and ebooks, at least in the public library area" (Ferrari, 2011, p. 9). The researchers concluded, "Some libraries would like ... to push away from journal/serial databases and look into online resources that are more service-based, namely those that provide language learning, job help, and car repair assistance" (Ferrari, 2001, p. 3).

This study set out to identify the core reference materials that constitute the common denominator for all reference professionals and that students graduating from programs in LIS should be familiar with. By selecting a panel of experts who work across geographic boundaries and in a variety of library settings, the final list reflects a basic toolkit that does not consider local needs, academic emphases of a particular LIS program, or specific workplace needs, and as such will likely be supplemented by LIS instructors. Nonetheless, the list will assist reference instructors and provide a compass in the dense landscape of reference sources.

Appendix 1: Profile of Delphi Experts [listed names are pseudonyms]

Jessica is a reference librarian with over 10 years' experience in an urban public university with a specialization in social sciences and business and which is a Master's L institution.

Carla is a multi-subject specialized librarian with over 10 years' experience (of which the last 5 have included teaching reference) in an urban private liberal arts college with a specialization in the humanities and social sciences and which is a Bac/A&S institution.

Liz is an Associate Professor and Global Studies librarian with over 10 years' experience in an urban public university with a broad range of arts and sciences specialties and which is a RU/VH institution.

Matt is a general librarian, who has taught reference, with over 10 years' experience in an urban public research library.

Kris is a general arts librarian, who teaches reference, with over 10 years' experience in an urban private university library with a broad range of arts and sciences specialties and which is a RU/VH institution.

Leslie is a general and reference librarian with about 5 years' experience in an urban vocational technical college library, which is a Bac/Assoc institution.

Shane is a general and reference librarian with over 10 years' experience in a private specialized university library with an emphasis on the arts and which is a Master's L institution.

Jen is an acquisitions and reference librarian with about 5 years' experience in an urban public library, this branch of which is located in an affluent neighborhood.

Ellen is a health sciences librarian with over 10 years' experience in the medical school library of an urban private university which is a Spec/Med institution.

Toni is a reference and government documents librarian with over 10 years' experience in an urban private university with a broad range of arts and sciences specialties and which is a RU/VH institution.

Claire is a senior reference and government documents librarian with over 10 years' experience in a suburban private university with a broad range of arts and sciences specialties and which is a RU/VH institution.

Jo is a reference librarian with a specialty in the arts with about 5 years experience in an unban research public library.

Appendix 2: Pre-test Questions Regarding Professional Expertise

1. In the past five years, have you taught a general reference course at a Library and Information Science program? [yes/no]

2. Are you now, or have you been in the past five years, a reference librarian? [yes/no]

3. Briefly, tell us about your professional experience.

For example:

"I am a reference librarian in a community college with 5 years experience"

"I am a social science librarian in a research intensive academic library"

"I am a public librarian and teach reference in a LIS distance program" [open ended]

Appendix 3: Pre-test Sources that Rank as Very Important by All Panelists

WorldCat and FirstSearch OCLC

Web of Science Citation Index



Library Literature

MLA International Bibliography

Oxford Dictionary of National Biography

Oxford English Dictionary (OED)

Project Muse

Dissertation Abstracts (ProQuest)

ERIC (Educational Resources Information Center)

Biography in Context (Gale, formerly

Biography Resource Center)

Chronicle of Higher Education

Gale Virtual Reference Library

History Resource Center (Gale)



Appendix 4: Results from Final Round (alphabetical)

19th Century Masterfile: Yes (64%) Academic Search Premier (EBSCO): Yes (100%)

Agricola: No (55%)

Almanac of New York City (Jackson,

Kenneth T & Kameny Fred, Eds.): No (80%)

Alternative Press Index: Yes (64%)

American Factfinder (U.S. Census Bureau): Yes (100%)

American National Biography (Oxford Univ.): Yes (100%)

Ancestry Library Edition: Yes (70%)

ARBA (American Reference Books Annual): Yes (59%)

ArchiveGrid (formerly RLG Archival Resources): Yes (70%)

arXiv (Cornell University): No (73%)

Bartlett's Familiar Quotations: Yes (100%)

Bibliographic Index Yes (50%) No (50%)

Biography and Genealogy Master Index: Yes (67%)

Biography in Context (Gale, formerly Biography Resource Center): Yes (67%)

Black Firsts: 2,000 Years of Extraordinary Achievement: Yes (55%)

Bloomberg: No (75%)

Booklist (reviews from ALA): Yes (70%)

Books in Print (Bowker): Yes (100%)

Bowker Annual Library & Book Trade Information: Yes (91%)

British Library General Catalog of Printed Books: No (82%)

Business & Company Resource Center (Gale): Yes (64%)

Business Source Premier (EBSCO): Yes (100%)

Cambridge Factfinder: No (100%)

Career Cruising: No (82%)

Chronicle of Higher Education: Yes (100%)

CIA World Factbook: Yes (100%)

College Blue Book: Yes (64%)

Columbia Gazetteer of the World: Yes (92%)

Columbia Granger's World of Poetry: Yes (82%)

Congressional Serial Set: Yes (50%) No (50%)

Contemporary Authors (Gale): Yes (70%)

Country Studies (Library of Congress): Yes (60%)

Credo Reference: Yes (73%) Yes (100%)

Dictionary of Literary Biography (Gale): Yes (67%)

Directories in Print (Gale): No (73%)

Directory of Publications and Broadcast Media (Gale): Yes (55%)

Directory of Special Libraries and Information Centers (Gale): No (70%)

Dissertation and Theses (ProQuest): Yes (100%)

Dun & Bradstreet's: Yes (55%)

EBSCOhost: Yes (100%)

Economist's Country Briefings: Yes (55%)

Emerald Management Xtra: Yes (55%)

Encyclopedia Americana: Yes (100%)

Encyclopedia Britannica: Yes (100%)

Encyclopedia Judaica: Yes (73%)

Encyclopedia of Library & Information Science (CRC Press): Yes (64%)

Encyclopedia of Religion: Yes (91%)

EndNoteWeb: Yes (64%)

ERIC (Educational Resources Information Center): Yes (91%)

Ethnic Newswatch: Yes (82%)

Europa World Yearbook: Yes (60%)

Europa: Gateway to the European Union: Yes (70%)

Europa World of Learning: Yes (64%)

Factiva (formerly Dow Jones Interactive): Yes (55%)

Facts on File World News Digest Yearbook: No (82%)

FDSys (GPO): Yes (100%)

Foundation Directory Online Professional: Yes (82%)

FRUS Foreign Relations of the United States: Yes (50%) No (50%)

Gale Virtual Reference Library: Yes (91%)

Genreflecting Advisory (print series): No (82%)

Getty Thesaurus of Geographic Names: Yes (64%)

Graphic Novels: A Genre Guide to Comic Books, Manga, and More: No (60%)

Grolier Online: No (90%)

Guide to Reference Books: Yes (91%)

Guinness World Records: Yes (64%)

HeritageQuest Online: No (90%)

Historical Abstracts (EBSCO): Yes (64%)

History Resource Center (Gale): Yes (64%)

Information Please Almanac: No (55%)

JSTOR: Yes (100%)

Justia: No (91%)

Kister's Best Encyclopedia: No (83%)

LearningExpress Library: No (82%)

Left Index: Yes (55%)

LexisNexis Digital Congressional Hearings Collection: Yes (50%) No (50%)

Library Literature Yes (100%)

Library of Congress Z39.50 gateway to catalogs: No (55%)

Literary Marketplace (Bowker): Yes (73%)

Mango Languages: No (91%)

MasterFILE Premier (EBSCO): Yes (92%)

Masterplots (Salem Press): No (60%)

McGraw-Hill Encyclopedia of Science & Technology: Yes (82%)

MEDLINE (EBSCO): Yes (100%)

MLA Directory of Periodicals: Yes (55%)

MLA International Bibliography: Yes (80%)

National Union Catalog (Library of Congress): Yes (91%)

New Dictionary of Cultural Literacy: No (91%)

New Walford Guide to Reference Resources: No (55%)

New York Times (Gale, ProQuest): Yes (100%)

Nonfiction: Selection Guide to Reference Books & Adult Nonfiction Yes (64%)

Official Museum Directory: No (82%)

Oxford Dictionary of National Biography: Yes (88%)

Oxford English Dictionary (OED): Yes (100%)

Oxford Reference Online: Yes (100%)

Perry-Castaneda Library Map Collection: Yes (64%)

Poole's Index to Periodical Literature (1802-1906): Yes (55%)

Project Muse: Yes (100%)

PsycINFO: Yes (82%)

PubMed (NLM/NIH): Yes (100%)

Reader's Guide: Yes (100%)

Reference USA: Yes (55%)

RefWorks: Yes (82%)

ScienceDirect (Elsevier): Yes (67%)

Scopus: Yes (82%)

Social Explorer: Yes (50%) No (50%)

SSRN (Social Science Research Network): Yes (82%)

Standard Periodical Directory: No (64%)

Stanford Encyclopedia of Philosophy: Yes (82%)

Stateman's Yearbook: Yes (91%)

Statistical Abstracts of the United States: Yes (100%)

Statistical Universe (ProQuest, formerly LexisNexis): Yes (80%)

Style manuals (APA, MLA, The Blue book, etc.): Yes (100%)

The Internet Archive: Yes (100%)

The Merck Manual Home Edition: Yes (91%)

The Reader's Advisor (R.R. Bowker): Yes (80%) Yes (100%)

Time Almanac: Yes (55%)

Times Atlas of the World: Yes (82%)

Times Atlas of World History: No (55%)

Times Comprehensive Atlas of the World: Yes (83%)

Today's Front Pages: No (91%)

Ulrich's Periodical Directory: Yes (100%)

United Nations Official Documents System: Yes (78%)

United Nations Treaty Series: No (70%)

Web of Science: Yes (92%)

Webster's Dictionary: Yes (100%)

Whitaker's Almanack: Yes (55%)

Wilson Biography Index: Yes (50%) No (50%)

World Almanac and Book of Facts: Yes (73%)

WorldCat and FirstSearch (OCLC): Yes (100%)

Zotero: Yes (91%)


The author wishes to express her gratitude to the twelve experts who volunteered their time and expertise to this study. In accordance with Delphi protocol, their identities remain anonymous.


Adkins, D., & Erdelez, S. (2006). An exploratory survey of reference source instruction in LIS courses. Reference and User Services Quarterly, 46(2), 50-60.

Agosto, D. E., Abels, E. G., Rozaklis, L., & MacDonald, C. The future of reference and information services in a virtual world. Juried poster presented at the 2009 Annual Meeting of the American Society for Information Science & Technology (ASIS&T), 6-11 November 2009. Vancouver, BC, Canada.

Agosto, D. E., Rozaklis, L., MacDonald, C., & Abels, E. G. (2010). Barriers and challenges to teaching reference in today's information environment. Journal of Education for Library and Information Science, 51(3), 177-186.

Bronstein, J., & Aharony, N. (2009). Views and dreams: A Delphi investigation into library 2.0 applications. Journal of Web Librarianship, 3(2), 89-109.

Du, Y. (2009). Librarians' responses to "reading at risk": A Delphi study. Library and Information Science Research, 31(1), 46-53.

Feret, B., & Marcinek, M. (1999). The future of the academic library and the academic librarian: A Delphi study. Librarian Career Development, 7(10), 91-107.

Feret, B., &, Marcinek, M. (2005). The future of the academic library and the academic librarian: A Delphi study reloaded. New Review of Information Networking, 11(1), 37-63.

Ferrari, A. (2011). Electronic resources for library staff: A survey by the Washington State Library. Retrieved from library/libraries/projects/sdl/Electronic%20Resources%20for%20Library%20Staff%20-%20 Survey%20Report%20and%20Analysis%20 -%202010.pdf

Fischer, R. (1978). The Delphi method: A description, review, and criticism. The Journal of Academic Librarianship, 4, 67-70.

Hsu, C. C., & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research & Evaluation, 12(10). Retrieved from asp?v=12&n=10

Kochtanek, T. R., & Hein, K. K. (1999). Delphi study of digital libraries. Information processing and Management, 35, 245-254.

Luo, L., & Wildemuth, B. M. (2009). Delphi studies. In B. M. Wildemuth, Applications of social research methods to questions in Information and Libray Science (pp. 83-92). Westport, CT: Libraries Unlimited.

Missingham, R. (2011). Parliamentary library and research services in the 21st century: A Delphi study. IFLA Journal, 37(1), 52-61.

Mon, L., Abels, E. G., Agosto, D. E., Japzon, A., Most, L., Masnik, M., & Hamann, J. (2008). Remote reference in U.S. public library and LIS education. Journal of Education for Library and Information Science, 49(3), 180-194.

Mon, L., & Randeree, E. (2009). On the boundaries of reference services: Questioning and library 2.0. Journal of Education for Library and Information Science, 50(3), 164-175.

Powell, R, R., & Connaway, L. S. (2004). Basic research methods for librarians. Westport, CT: Libraries Unlimited.

RUSA Guidelines for Behavioral Performance of Reference and Information Service Providers (2004). Retrieved from resources/guidelines/guidelinesbehavioral

Shaw, D., & Okada, E. (2001). A collaborative approach to teaching "reference". Indiana Libraries, 23(1), 41-15.

Westbrook, L. (1997). Information access issues for interdisciplinary scholars: Results of a Delphi study on women's studies research. The Journal of Academic Librarianship, 23(3), 211-216.

Debbie Rabina

Pratt Institute, School of Information and Library Science, 144 West 14th Street, 6th fl., New York, NY, 10011-7301.


Table 1. Items that Received 100 +/- Consensus. ([+] Describes
positive consensus and [-] describes negative consensus).

Source                               100 (+/-)         Category

Oxford Reference Online                  +           Compilations
WorldCat & FirstSearch                   +             Catalogs
Ulrich's Periodical Directory            +            Tools for
Library Literature                       +            Tools for
Oxford English Dictionary                +           Dictionaries
Webster's Dictionary                     +           Dictionaries
Bartlett's Familiar Quotations           +           Dictionaries
Encyclopedia Americana                   +          Encyclopedias
Encyclopedia Britannica                  +          Encyclopedias
Business Source Premier                  +           Directories-
Academic Search Premier                  +         Indexes-General
ProQuest Dissertations and Theses        +         Indexes-General
EBSCOHost                                +         Indexes-General
JSTOR                                    +         Indexes-General
Project Muse                             +         Indexes-General
Reader's Guide                           +         Indexes-General
The Internet Archive                     +         Indexes-General
Medline (EBSCO)                          +         Indexes-General
PubMed (NLM/NIH)                         +         Indexes-General
Chronicle of Higher Education            +            Tools for
The New York Times                       +            Newspapers
Style manuals (APA, MLA, etc.)           +          Bibliographic
                                                    citations tools
Books in Print                           +          Bibliographies
American Factfinder                      +       Statistical sources                                 +       Statistical sources
Statistical Abstracts of the U.S.        +       Statistical sources
American National Biography              +       Biographical sources
CIA World Factbook                       +           Government/
                                                      NGO sources
FDsys (formally GPO Access)              +           Government/
                                                      NGO sources                               +           Government/
                                                      NGO sources
The Cambridge Factfinder                 -         Ready reference

Table 2. Items that Received High Agreement (+/- 80-99%).
([+] Describes positive consensus and [-] describes negative

Source                        90-99   80-89         Category
                              (+/-)   (+/-)

Gale Virtual Reference          +                 Compilations
National Union Catalog          +                   Catalog
Merck Manual Home Edition       +               Ready reference
Stateman's Yearbook             +               Ready reference
Bowker Annual                   +             Tools for librarians
Guide to Reference              +             Tools for librarians
Encyclopedia of Religion        +                Encyclopedias
Columbia Gazetteer              +               Geographic tools
  of the World
ERIC                            +               Indexes-General
MasterFile Premier (EBSCO)      +               Indexes-General
Web of Science                  +               Citation indexes
Zotero                          -                Bibliographic
                                                  citation tools
Grolier Online                  -                 Compilations
Mango Languages                 -                 How-to ready
New Dictionary of               -                 Dictionaries
  Cultural Literacy
Today's Front Pages             -                  Newspapers
Heritage Quest Online           -                 Biographies
Justia                          -                 Government/
                                                   NGO sources
Columbia Granger's                      +       Indexes-General
  World Poetry
PsycINFO                                +       Indexes-General
SSRN (Social Science                    +       Indexes-General
  Research Network)
Ethnic Newswatch                        +          Newspapers
Scopus                                  +       Citation indexes
RefWorks                                +        Bibliographic
                                                  citation tools
MLA International                       +        Bibliographies
The Reader's Advisor                    +      Reader Advisories
Statistical Universe                    +     Statistical sources
Oxford Dictionary of                    +         Biographies
  National Biography
British Library General                 -           Catalog
  Catalog of Printed Books
Almanac of the City of NY               -       Ready reference
Facts on File                           -       Ready reference
Career Cruising                         -         How-to ready
LearningExpress Library                 -         How-to ready
Kister's Best Encyclopedias             -        Encyclopedias
Genreflecting Advisory                  -      Reader Advisories
Official Museum Directory               -         Directories

Figure 3. Most popular resources at New
York Public Library 1/23/12. From http://


Freegal Musics
Literature Resource Center
Mango Languages
Berg Fashion Library
Oxford Reference Online
Biography in Context
Business Source Premier
Ancestry Library Edition
COPYRIGHT 2013 University of Toronto Press
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2013 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:library and information science
Author:Rabina, Debbie
Publication:Journal of Education for Library and Information Science
Article Type:Report
Geographic Code:1USA
Date:Mar 22, 2013
Previous Article:Testing the benefits of blended education: using social technology to foster collaboration and knowledge sharing in face-to-face LIS courses.
Next Article:A case study of periodical use by library and information science students.

Terms of use | Privacy policy | Copyright © 2022 Farlex, Inc. | Feedback | For webmasters |