Evaluating reference services in the electronic age.
IN AN ELECTRONIC ERA, THE EVALUATION OF REFERENCE and related information services See Information Systems. should still be based on the same principles used to evaluate traditional face-to-face (jargon, chat) face-to-face - (F2F, IRL) Used to describe personal interaction in real life as opposed to via some digital or electronic communications medium. reference services and printed reference tools. Traditional research methods--which are surveys and questionnaires, observation, individual and focus group interviews, and case studies--can be utilized very effectively in an electronic environment. However, electronic technologies offer interesting research opportunities not present in the traditional reference environment.
At conferences and workshops on evaluating reference services, the most frequent recurring re·cur
intr.v. re·curred, re·cur·ring, re·curs
1. To happen, come up, or show up again or repeatedly.
2. To return to one's attention or memory.
3. To return in thought or discourse. question librarians This is a list of people who have practised as a librarian and are well-known, either for their contributions to the library profession or primarily in some other field. ask is, "How can the material on evaluating reference services be applied to assessing electronic reference services?" The best answer is, "Take existing methods, determine which will best meet the study goals, and then adapt those methods to the electronic environment."
In any environment, evaluating reference services still requires starting by assessing why reference services are being evaluated and what the organization plans to do with the study results. Before trying to decide how to evaluate electronic services, performance standards that set the level of achievement expected for the service should be explicitly stated. In determining the performance standards to be adopted, the organization must decide what values are crucial. Are members of the organization concerned primarily with
1. Economics--the cost or productivity of services;
2. The process--aspects of librarian/reference system and user interaction;
3. Resources--books, indexes, databases, staffing levels, equipment, design of physical or electronic environment; or
4. Products/outcomes--information or knowledge that the users obtain.
In an electronic environment the interactions between librarians and users often will no longer be truly face to face. Thus, process standards are the measures that most need to be reviewed in a digital reference environment. Librarian (1) A person who works in the data library and keeps track of the tapes and disks that are stored and logged out for use. Also known as a "file librarian" or "media librarian." See data library.
(2) See CA-Librarian. behaviors that are crucial in the reference-desk environment will need to be redefined for remote reference services. Work on redefining process standards has already begun. The Virtual Reference Desk (VRD VRD Virtual Reference Desk
VRD Virtual Retinal Display
VRD Voirie et Réseaux Divers (French: External Works, or Roads & Utility Services)
VRD Vocational Rehabilitation Division (Oregon DHS) ) project has developed a list of User Transaction Standards to address aspects of librarian/system and user interaction. The standards address several "facets" related to quality: accessible, prompt turnaround Turnaround
A situation where a company that has had poor performance for an extended period of time experiences a positive reversal.
A speculator may profit from a turnaround if he or she accurately anticipates the improvement of a poorly performing company. , clear response policy, interactive and instructive in·struc·tive
Conveying knowledge or information; enlightening.
in·structive·ly adv. (Kasowitz, Bennett & Lankes, 2000). Most of these facets address the process standards, rather than standards related to economics, resources, or products/outcomes.
In a remote electronic reference environment, accessibility and prompt turnaround could become dominant in user evaluations. Miwa (2000) used digital reference services Digital reference is a service by which library reference service is conducted online, and the reference transaction is a computer-mediated communication.
The word "reference" in this context refers to the task of providing assistance to library users in finding information, features of acknowledgment acknowledgment, in law, formal declaration or admission by a person who executed an instrument (e.g., a will or a deed) that the instrument is his. The acknowledgment is made before a court, a notary public, or any other authorized person. , responsiveness, and tone of message to represent the process aspects of the reference interaction in a digital environment. She also looks at user situations as part of the process--for example, wording of the request by the user and user's ability to comprehend the message.
Broad goals for the study should be prepared in writing once a reasonable degree of consensus has been achieved on the particular set of standards that an organization wishes to emphasize. After broad goals have been developed, written objectives should be developed for each study goal. The objectives should be measurable so that, at the conclusion of the evaluation, one can identify any gaps between the present level and the desired level of reference service performance.
This present paper discusses how to apply traditional evaluation methods in an electronic reference environment once the study goals and objectives have been determined. Readers desiring additional information on setting performance standards and developing goals and objectives for reference service evaluation may wish to consult Evaluating Reference Services: A Practical Guide (Whitlatch, 2000).
All methods have strengths and weaknesses. Depending on the goals and objectives of the study, some methods will be more effective than others. As a general rule, utilizing more than one method is recommended in a single study, because the strengths of one method often compensate for the weaknesses of another. The advantages and disadvantages of the various methods may also change somewhat in an electronic environment. This paper considers how applying surveys, observation, interviews, and case studies--all traditional evaluation methods used in assessing face-to-face services--presents new opportunities and challenges in assessing electronic reference services.
SURVEYS AND QUESTIONNAIRES
Surveys or questionnaires are methods of directly collecting information on individuals' thoughts, beliefs, attitudes, and opinions, plus objective data, such as education, gender, and income. The survey method has been the most frequent way of assessing traditional reference services. In the past, surveys have been relied upon too heavily because they are the most efficient method of assessing a large group of representative users. Also. for the inexperienced in·ex·pe·ri·ence
1. Lack of experience.
2. Lack of the knowledge gained from experience.
in researcher, surveys appear easy to design. The disadvantages, such as obtaining meaningless information from poorly designed questions and the lack of depth of information from standardized standardized
pertaining to data that have been submitted to standardization procedures.
standardized morbidity rate
see morbidity rate.
standardized mortality rate
see mortality rate. responses, are often not appreciated until too late. Another significant problem in using surveys is low response rates, particularly from surveys distributed through the mail. A substantial number of nonrespondents can bias the results; those who choose not to complete the survey might hold very different views from those who do.
Publicly accessible computer network connecting many smaller networks from around the world. It grew out of a U.S. Defense Department program called ARPANET (Advanced Research Projects Agency Network), established in 1969 with connections between computers at the questionnaires can be used effectively to survey attitudes and opinions on the quality of reference service related to process (the interaction with the virtual reference service) and products/outcomes (the value of the information obtained). An Internet survey asking for an evaluation of service provided can be sent out within days after the user has received an answer. In contrast to surveys distributed in person at the reference desk or in the library, emailing the questionnaire can also be calculated to allow most users some time to use and further evaluate the information obtained through a reference interaction.
As Zhang (1999) points out, the Web provides new opportunities to conduct survey research more efficiently. Research costs for sending out Internet surveys are relatively low and the turn-around time short compared to conventional mail-in surveys. Also, email can be used effectively to follow up on paper-based surveys (Roselle Roselle (rōzĕl`), borough (1990 pop. 20,314), Union co., NE N.J.; set off from Linden 1890 and inc. 1894. Chiefly residential, the borough has some industry. & Neufeld, 1998). Most responses received in electronic format have been precoded, eliminating transcription errors A transcription error is a specific type of data entry error that is commonly made by human operators or by optical character recognition programs (OCR). Human transcription errors are commonly the result of typographical mistakes, putting fingers in the wrong place during touch and saving time and expense. McCullough Mc·Cul·lough , David Born 1933.
American historian who was awarded a Pulitzer Prize for his biography of Harry Truman, Truman (1992). (1998) notes that Web-based surveys are faster, generate more accurate information, and cost less. He has found that a respondent In Equity practice, the party who answers a bill or other proceeding in equity. The party against whom an appeal or motion, an application for a court order, is instituted and who is required to answer in order to protect his or her interests. will typically complete a Web-based survey in about half the time it would take an interviewer to conduct that survey by telephone or in person.
Resolving the technical problems with Internet surveys requires greater technical expertise on the part of the researcher than does research conducted with traditional survey methods. However, services that provide Web survey forms and guidance to assist researchers in designing and developing Internet surveys are becoming common. Names and Web addresses for some of the services that have been positively discussed on the Academy of Management Research Listserv, firstname.lastname@example.org See .edu.
(networking) edu - ("education") The top-level domain for educational establishments in the USA (and some other countries). E.g. "mit.edu". The UK equivalent is "ac.uk". , are provided in the Appendix.
Zhang (1999) also reviews potential problems and concerns related to Internet-based surveys. One of the greatest strengths of survey research is the ability to randomly select respondents In the context of marketing research, a representative sample drawn from a larger population of people from whom information is collected and used to develop or confirm marketing strategy. in a manner that ensures a sample representative of the target population. In telephone surveys, respondents are randomly selected, but most online poll respondents are self-selected (Pew Research Center The Pew Research Center is a "fact tank" based in Washington, D.C., that provides information on the issues, attitudes and trends shaping the USA and the world. The Center and its projects receive funding from The Pew Charitable Trusts. , 1999). The greatest difficulties with Internet surveys occur when the survey does not reach certain types of respondents who need to be included in the survey population. Biased samples A biased sample is a statistical sample of a population where some members of the population are less likely to be included than others. An extreme form of biased sampling occurs when certain members of the population are totally excluded from the sample (that is, they have zero and returns can be a major problem because certain social groups are underrepresented un·der·rep·re·sent·ed
Insufficiently or inadequately represented: the underrepresented minority groups, ignored by the government. among Internet users Internet user n → internauta m/f
Internet user Internet n → internaute m/f .
However, for surveying users of electronic reference services, bias should be minimal. Respondents must have access to the Internet in order to use the electronic services; they can presumably pre·sum·a·ble
That can be presumed or taken for granted; reasonable as a supposition: presumable causes of the disaster. access a Web survey form as well. Some individuals may not have convenient access from their home or office and may use the service only occasionally from an Internet cafe The high-tech equivalent of the coffee house. However, instead of playing chess or having heated political discussions, you browse the Internet and discuss the latest technology. CDs, DVDs, games and other "cyber stuff" are also generally available. or a library. If these individuals are not identified, this group may be underrepresented. Individuals who do not have convenient access may, as a whole, be less experienced users of electronic reference services. If these users are not included in the sample, survey results may not truly represent the population as a whole. Other means, such as a telephone interview or mail survey, may be required to obtain responses from them. Finally, if the purpose of the survey is to collect information from people who do not use electronic sources, relying upon the Internet as the principal method of survey delivery will present a very serious problem.
In addition, low response rates are a serious problem with Internet surveys. In her evaluation of AskERIC, Shostack (2000) observed that users were either extremely happy or dissatisfied dis·sat·is·fied
Feeling or exhibiting a lack of contentment or satisfaction.
dis·satis·fied with digital reference services. These results suggest that only motivated mo·ti·vate
tr.v. mo·ti·vat·ed, mo·ti·vat·ing, mo·ti·vates
To provide with an incentive; move to action; impel.
mo users are responding. A study that replicated an earlier study found a disturbing decline in email response rates: in 1995 the email response rate was 80 percent, but by 1998 it had fallen to 42 percent (Bachmann Bachmann may refer to: People
Zhang (1999) concludes that the Internet cannot serve as the only means to collect survey data if researchers need representative returns from a sample. Schaefer Schäfer is German language word for shepherd.
It is also a common surname, alternatively spelled Schaefer or Schaeffer (or anglicised to Shafer or Shaffer). and Dillman Dillman is a family name or surname.
In order to apply scientifically tested polling techniques to Internet technologies, Stanford political scientists Douglas Rivers and Norman Nie have created Knowledge Networks. With $42 million in venture capital, they have installed free WebTV devices normally costing $250 each in 40,000 homes selected through random phone calls. Because everyone in the household nineteen or older is involved, there are about 100,000 participants. The homes receive a black box slightly smaller than a VCR VCR: see videocassette recorder.
in full videocassette recorder
Electromechanical device that records, stores on a videotape cassette, and plays back on a TV set recorded images and sound. , a cordless cord·less
Having no cord, usually using batteries as a source of power: a cordless telephone.
cord keyboard, and many instructions. The homes are expected to remain in the survey pool for three years. In exchange for answering brief surveys about once a week, the households receive free Internet access See how to access the Internet. , email, and frequent chances to win prizes. Of those who were asked to join the Knowledge Networks pool, 56 percent agreed--compared with 15 percent of people who usually agree to participate in phone polls. Although the polling is a significant activity, the primary company income comes from consumer research for manufacturers (Konigsmark, 2000).
Zhang (1999) also reports that validity of Internet survey responses can be adversely affected. Unintended participants may respond because of the ease of forwarding email messages to other people. Individuals can respond to a single survey by submitting the same reply many times. Unique case-identification numbers should be assigned as·sign
tr.v. as·signed, as·sign·ing, as·signs
1. To set apart for a particular purpose; designate: assigned a day for the inspection.
2. to each respondent to control for multiple responses and unintended participants.
Nondeliverable surveys are also a major disadvantage of email. In 1995 and 1998 studies, Bachmann, Elfrink and Vazzana (1999) found that about 20 percent of all emailed surveys were nondeliverable.
Comfort level with the Internet survey form should also be considered. Zhang (1999) found that, while 80 percent of usable USable is a special idea contest to transfer US American ideas into practice in Germany. USable is initiated by the German Körber-Stiftung (foundation Körber). It is doted with 150,000 Euro and awarded every two years. replies were received via the Web, 20 percent of respondents chose to complete the survey via postal mail or fax. Internet survey respondents did report problems with the layout of the survey questionnaire on low-resolution monitors, problems going back to previous parts of the questionnaire, problems with printing, and (on computers with low-speed modems) problems with downloading downloading - download the questionnaire. Users also reported that comments were also more difficult to insert on electronic survey forms than on paper forms. Shostack (2000) also noted a tendency for users to ignore open-ended questions A closed-ended question is a form of question, which normally can be answered with a simple "yes/no" dichotomous question, a specific simple piece of information, or a selection from multiple choices (multiple-choice question), if one excludes such non-answer responses as dodging a on Internet survey forms. (This problem is not unique to online surveys. In the author's experience, most users completing paper forms also tend to leave open-ended questions blank.) Surveys not conducted by telephone or in-person interview tend to have rather limited potential to collect qualitative data. An experiment with incentives in the form of cash prizes revealed that, while the overall numbers of respondents did not increase significantly, the number of completed Internet survey questionnaires did rise (Pitkow & Kehoe, 1996).
Observational methods collect information on people as they behave in real-life situations. Forms of observation that have been used to assess the quality of reference services include direct observation of the reference interview, observers disguised dis·guise
tr.v. dis·guised, dis·guis·ing, dis·guis·es
a. To modify the manner or appearance of in order to prevent recognition.
b. To furnish with a disguise.
2. as patrons asking preassigned questions, self-observation in the form of diaries or journals, recording interviews with audio or videotape videotape
Magnetic tape used to record visual images and sound, or the recording itself. There are two types of videotape recorders, the transverse (or quad) and the helical. , reviewing data collected as part of daily library operations, and examining information on reference transactions collected for another purpose.
Observational methods have been less frequently used than surveys to evaluate reference services, because this method requires a greater investment of staff time. Safeguarding against observational bias also requires training observers thoroughly and may require using more than one observer.
The electronic reference service environment offers some new and exciting opportunities in use of observational methods. Information on electronic reference transactions can be collected and archived as part of ongoing library operations much more easily than can information on traditional reference interviews. Content analysis of these electronic questions should enable us systematically to study the nature of the questions, sources used, and skills required to a much greater extent than is possible in face-to-face reference interactions. The review and analysis of samples from archives of questions and answers provide a practical tool to diagnose diagnose /di·ag·nose/ (di´ag-nos) to identify or recognize a disease.
1. To distinguish or identify a disease by diagnosis.
2. problems and improve services.
Studies of email reference questions that use observational techniques In marketing and the social sciences, observational research (or field research) is a social research technique that involves the direct observation of phenomena in their natural setting. are already underway. Garnsey and Powell (2000) examined and classified email reference questions into one of the following categories based on content: (1) ready reference; (2) research question; (3) genealogy genealogy (jē'nēŏl`əjē, –ăl`–, jĕ–), the study of family lineage. Genealogies have existed since ancient times. ; (4) library technology; (5) request for materials; (6) bibliographic bib·li·og·ra·phy
n. pl. bib·li·og·ra·phies
1. A list of the works of a specific author or publisher.
a. verification; and (7) other. Jones, Carter, and Memmott (1999) used a random sample of academic libraries to study the proportion of libraries offering digital reference services and to examine the characteristics of those services. They looked at size of libraries, direct links from library home pages, ways in which users were able to submit questions, FAQ (Frequently Asked Questions) A group of commonly asked questions about a subject along with the answers. Vendors often display them on their Web sites for use as troubleshooting guidelines. documents, policies, institutional barriers, and the role of type of institutional funding (public vs. private). Shostack (2000) analyzed an·a·lyze
tr.v. an·a·lyzed, an·a·lyz·ing, an·a·lyz·es
1. To examine methodically by separating into parts and studying their interrelations.
2. Chemistry To make a chemical analysis of.
3. questions that had been submitted via a question submission form to AskERIC. She found that over 80 percent of users filled out the form completely. Staff were also asked to change the subject line of the response to the topic of the reference query so that questions could be classified by topic.
However, the ease of collecting such information does raise the level of concern about protecting the individual's rights to privacy. The first rule of ethics ethics, in philosophy, the study and evaluation of human conduct in the light of moral principles. Moral principles may be viewed either as the standard of conduct that individuals have constructed for themselves or as the body of obligations and duties that a in research is to do no harm to the participants. In using data for research, particular attention must be paid to protecting the identity of individual users when archiving questions and answers. Access should be restricted to all information that might reveal people's identities. Names and specific information that have the potential to identify individual participants, such as physical descriptions, very detailed demographic information, or identifying events or places, should be removed or modified. Without proper protections, publication of the analysis could harm the morale and self-esteem self-esteem
Sense of personal worth and ability that is fundamental to an individual's identity. Family relationships during childhood are believed to play a crucial role in its development. of reference librarians, staff, and users.
Gray (2000) used observational methods to analyze Web sites of ten large research libraries that provide virtual reference services. The approaches to centralization cen·tral·ize
v. cen·tral·ized, cen·tral·iz·ing, cen·tral·iz·es
1. To draw into or toward a center; consolidate.
2. , placement of the link to reference services on the Web page, use of forms, definition of client base, response times, and question types accepted were analyzed. Observational methods are also useful for testing the effectiveness of different types of answering sources. To compare the effectiveness of print and paper-based reference sources in answering different types of reference questions, Havener (1990) divided 68 reference librarians into two different groups. Members of one group were permitted to use only print tools in their research, while members of the other group could use only online sources to answer the same set of questions. Information recorded varied by question type--for conceptual questions, librarians were asked to record ten relevant citations; for factual questions, librarians were asked to provide only one relevant fact. Time spent was also recorded. In an exploratory study, Janes and McClure (1999) compared the accuracy of answers found in freely available Web sites and traditional print-based sources by asking participating librarians and library school students to answer 12 questions only with resources they were directed to use (either Web or non-Web). Connell and Tipple (1999) gathered ready reference questions that were actually asked by users over a two-week period and then, using AltaVista as a search engine, searched for and examined the accuracy of answers found on the Web.
Observational methods are useful in determining the difficulty that users encounter with online reference tools. Chisman, Diller and Walbridge (1999) advertised for volunteers who were paid ten dollars for their participation. A usability How easy something is to use. Both software and Web sites can be tested for usability. Considering how difficult applications are to use and Web sites are to navigate, one would wish that more designers took this seriously. See user interface and usability lab. test was designed to determine how easily users could navigate (1) "Surfing the Web." To move from page to page on the Web.
(2) To move through the menu structure in a software application. a Web catalog catalog, descriptive list, on cards or in a book, of the contents of a library. Assurbanipal's library at Nineveh was cataloged on shelves of slate. The first known subject catalog was compiled by Callimachus at the Alexandrian Library in the 3d cent. B.C. and whether they understood what they were seeing. Observers recorded the search strategy, comments made by the participants, observations about the participants' responses, success, and the time needed to complete the task.
Unobtrusive observation methods can also be used effectively in an electronic world. Reference questions can be prepared and answers determined for factual types of questions. Graduate students or others who are posing as users with questions can query both commercial and non-profit "ask a question" services. Results can be analyzed by such factors as response time, accuracy or quality of answer, tone of message, ease of submitting the question, and observations on whether people would return to the site again.
INDIVIDUAL INTERVIEWS AND FOCUS GROUP INTERVIEWS
Interviews are an appropriate method for collecting information on how people interpret their world, describe their experiences, and articulate articulate /ar·tic·u·late/ (ahr-tik´u-lat)
1. to pronounce clearly and distinctly.
2. to make speech sounds by manipulation of the vocal organs.
3. to express in coherent verbal form.
4. their attitudes, perspectives, concerns, and values. Despite the potential for gathering in-depth information, interviews have been less frequently used than surveys because of the expense and time required. As is the case with observational methods, interviewers must be thoroughly trained to avoid bias. The management and coordination of scheduling for either individual or group interviews can be extremely time consuming. Coding and analyzing the data also require considerable time.
Interviews of both users and librarians are also possible in the digital reference service environment. Interviewers can use Web-based survey forms to record the results of interviews efficiently. However, users will probably be harder to reach than in-person users of reference-desk services. Marketers have begun to use online focus groups; chat technology with these methods could certainly be adopted for users of electronic reference services. While online focus groups do not allow moderators to observe how people are interacting, benefits include no geographic barriers, lower costs, more rapid turn-around time, and the possibility that participants may be more open because of the greater anonymity provided by chat rooms (Maddox, 1998).
Conventional focus groups can also be used effectively to evaluate digital reference services. By reaching out to user groups in the community (teenagers at risk, small business organizations, etc.) or distance learning communities in an academic setting, participants can be recruited to assess their experience with digital reference services. Food or some other small gift of appreciation and a convenient location will encourage participation.
Case studies use a combination of assessment methods to analyze services in one or in a limited number of situations. Case studies have been used to assess new reference services or products. Combining the different methods will enrich study findings significantly, but will also increase the time required to conduct the study and analyze the information collected. Results generally cannot be applied to other situations.
Case studies have great potential to improve our understanding of the quality of digital reference services. Using information provision in a hospital setting, Barcellos (2000) is studying user intermediary Intermediary
See: Financial intermediary
See financial intermediary. interactions through use of organizational publications, site observations, transaction logs, and interviews of both users and intermediaries. A case study of the Internet Public Library Internet Public Library - (IPL) A project at the University of Michigan School of Information and Library Studies to provide an on-line, 24 hour public library, chaired by an assemblage of librarians and information industry professionals. Reference Division examined unanswered questions to determine why they were not being answered and to generalize generalize /gen·er·al·ize/ (-iz)
1. to spread throughout the body, as when local disease becomes systemic.
2. to form a general principle; to reason inductively. about the difficulties associated with providing reference services via the Internet (Ryan, 1996). White (1999) has developed a framework for evaluating electronic question-answering services that involves World Wide Web inspection, perusal of publicly available policy documents, and personal contact via email and/or interviews with service administrators.
Several years ago, James Rettig (1996) observed that many of the criteria used for evaluating printed reference resources have analogs in the digital world: for example, authority, accuracy, level or audience, and content. Standards and methods used for evaluating traditional reference services also have many analogs in the world of digital reference. Standards and criteria related to economic considerations, the reference process, reference resources, or products or service outcomes will still be important in an electronic world. Traditional methods of survey, observation, interview, and case study remain useful.
Case studies that focus on evaluating experimental digital reference services and employ a variety of research methods may have the greatest promise to enhance our knowledge. Case studies have the potential to improve our knowledge of both the effectiveness of digital reference services and the combination of methods best suited to evaluate them. Over time, the profession should, through the effective use of case studies, be able to build a guide to best practices, not only for digital reference services, but also for the methods necessary to assess and continually con·tin·u·al
1. Recurring regularly or frequently: the continual need to pay the mortgage.
2. improve these services.
Results of initial studies of digital reference services and the now well-known phenomenon of declining business at many reference desks also suggest that these studies should be used to analyze future directions in reference practice. Studies (Connell & Tipple, 1999; Janes & McClure, 1999) indicate that freely available Web materials may serve as well as traditional ready reference tools for answering many of the common types of queries received at reference desks. For most users, convenience is first. The expert in-person assistance a librarian might provide is becoming comparatively less convenient than it once was, when the alternative source is the Web. Many users will love the convenience and be satisfied with "good enough." Others will find it more convenient to take advantage of remote ready reference services, which will probably be supported by a relatively small amount of funding or reference librarians from each local library.
As the demand continues to shift away from the reference desk, libraries have the opportunity to establish much more active outreach Outreach is an effort by an organization or group to connect its ideas or practices to the efforts of other organizations, groups, specific audiences or the general public. programs. The public and administrators may come to view reference librarians as less essential than in past times. While libraries still have reference librarians, shifting patterns of user demands for reference services provide libraries with opportunities to emphasize different strategies to connect library materials with users. Libraries may develop a stronger role in the community in promoting information competencies through partnerships with community service agencies or, within the academic community, with faculty engaged in critical thinking and writing courses.
Changes in strategy would also have implications for professional education. Marketing skills that are essential for developing active outreach programs, as well as instructional skills, may need to become a major part of the core curriculum in every library school. One of the essential marketing skills is evaluation and improvement of outreach efforts. Perhaps the day will come when all librarians engaged in professional practice will receive, as part of their professional education, in-depth understanding and experience in developing and applying survey, observation, interview, and case-study methods so that reference librarians might change, survive and prosper in the new electronic age.
APPENDIX: SURVEY ASSISTANCE ON THE WEB Internet Survey Solutions http://www.clearpicture.com/Survey_Solutions.htm Web-based Clear Picture survey system. Research Internet Advertising Resource Guide http://www.admedia.org/internet/research.html Annotated entries for research firms, online surveys, virtual focus groups, survey software. Survey Select http://www.surveyselect.com/ Samples of the Saja software product available for viewing on the Web site. Zoomerang Create Surveys http://www.zoomerang.com/build_preview/new-survey.zgi?1182 Survey templates for business, community, personal/social, and education.
Bachmann, D. P.; Elfrink, J.; & Vazzana, G. (1999). E-mail and snail mail Mail sent via a country's government-regulated postal system.
(messaging) snail mail - (Or "snailmail", "smail" from "US Mail" via "USnail"; "paper mail"). Bits of dead tree sent via the postal service as opposed to electronic mail. face off in rematch REMATCH Cardiology Clinical trials–Randomized Evaluation of Mechanical Assistance Therapy as an alternative in Congestive Heart failure–related to use of a portable, electric left ventricular-assist system–LVAS–eg, HeartMate® . Marketing Research, 11(4), 10-15.
Barcellos, S. (2000). Understanding intermediation in a digital environment: An exploratory case study Paper presented at Facets of Digital Reference: The Virtual Reference Desk 2nd Annual Digital Reference Conference. Summary available from http://www.vrd.org/.
Chisman, J.; Diller, K.; & Walbridge, S. (1999). Usability testing Usability testing is a means for measuring how well people can use some human-made object (such as a web page, a computer interface, a document, or a device) for its intended purpose, i.e. usability testing measures the usability of the object. : A case study. College & Research Libraries, 60(6), 552-569.
Connell, T. H., & Tipple, J. E. (1999). Testing the accuracy of information on the World Wide Web using the AltaVista search engine. Reference & User Services Quarterly, 38(4), 360-368.
Garnsey, B. A., & Powell, R. R. (2000). Electronic mail reference services in the public library. Reference & User Services Quarterly, 39(3), 245-254.
Gray, S. M. (2000). Virtual reference services: Directions and agendas. Reference & User Services Quarterly, 39(4), 365-373.
Havener, W. M. (1990). Answering ready reference questions: Print versus online. Online, 14 (1), 22-28.
Janes, J.; Carter, D.; & Memmott, P. (1999). Digital reference services in academic libraries. Reference & User Services Quarterly, 39(2), 145-150.
Janes, J., & McClure, C. R. (1999). The Web as a reference tool: Comparisons with traditional sources. Public Libraries, 38(1), 30-39.
Kasowitz, A.; Bennett, B.; & Lankes, R. D. (2000). Quality standards for digital reference consortia. Reference & User Services Quarterly, 39(4), 355-363.
Konigsmark, A. R. (2000), High tech joins polls. San Jose Mercury News The San Jose Mercury News is the major daily newspaper in San Jose, California and Silicon Valley. The paper is owned by MediaNews Group. Its headquarters and printing plant are located in North San Jose next to the Nimitz Freeway (Interstate 880). , November 12 B1, B5.
Maddox, K. (1998). Virtual panels add real insight for marketers: Online focus-group use expanding. Advertising Age. Retrieved August 10, 2000 from http://adage.com/interactive/daily/archives/id199806.html.
McCullough, D. (1998). Web-based market research ushers in a new age. Marketing News, 32, 27-28.
Miwa, M. (2000). User situations in digital reference service: An evaluation of the AskERIC Q & A Service. Paper presented at Facets of Digital Reference: The Virtual Reference Desk 2nd Annual Digital Reference Conference. Summary of presentation from http:// www.vrd.org/.
Pew Research Center. (1999). A survey methods comparison: Online polling offers mixed results. Retrieved March 19, 2000 from http://www.people-press.org/onlinerpt.htm.
Pitkow, J., & Kehoe, C. (1996). GVU's 6th WWW User Survey. Paper posted on the World Wide Web. Retrieved January 25, 1997 from http://www.cc.gatech.edu/gvu/user_surveys/ survey-10-1996/#exec.
Rettig, J. (1996). Beyond "cool": Analog models for reviewing digital resources. Online. Retrieved December 30, 1998 from http://www.onlineinc.com/onlinemag/SeptOL/ rettig9.html.
Roselle, A., & Neufeld, S. (1998). The utility of electronic mail follow-ups for library research. Library and Information Science Research, 20(2), 153-161.
Ryan, S. (1996). Reference service for the Internet community: A case study of the Internet Public Library Reference Division. Library & Information Science Research, 18(3), 241-259.
Schaefer, D. R., & Dillman, D.A. (1998). Development of a standard e-mail methodology. Public Opinion Quarterly, 62(3), 378-379.
Shostack, P. L. (2000). Identifying users' needs. Paper presented at Facets of Digital Reference: The Virtual Reference Desk 2nd Annual Digital Reference Conference. Summary of presentation available from http://www.vrd.org.
Virtual Reference Desk. http://www.vrd.org/training/facets.html.
White, M. D. (Ed.) (1999). Analyzing electronic question/answer services: Framework and evaluations of selected services (Report No. CLIS-TR-99-02). College Park, MD: College of Library and Information Services, University of Maryland University of Maryland can refer to:
Whitlatch, J. B. (2000). Evaluating reference services: A practical guide. Chicago: American Library Association American Library Association, founded 1876, organization whose purpose is to increase the usefulness of books through the improvement and extension of library services. .
Zhang, Y. (1999). Using the Internet for survey research: A case study. Journal of the American Society for Information Science, 51(1), 57-68.
Jo Bell Whitlatch, Reference Librarian & History Selector (programming) selector - 1. In Smalltalk or Objective C, the syntax of a message which selects a particular method in the target object.
2. An operation that returns the state of an object but does not alter that state. , San Jose San Jose, city, United States
San Jose (sănəzā`, săn hōzā`), city (1990 pop. 782,248), seat of Santa Clara co., W central Calif.; founded 1777, inc. 1850. State University, San Jose CA 95192-0028
JO BELL WHITLATCH is Associate Dean at San Jose State University Library. She also teaches reference courses in the Graduate School of Library and Information Science A School of Library and Information Science (SLIS) is a university-based institution that provides a Master's degree or other advanced degrees associated with Library science, Information Science, or a combination of the two. at San Jose. She is a past president of the Reference User Services Association (RUSA RUSA Reference and User Services Association (American Library Association)
RUSA Randonneurs USA (endurance cycling)
RuSa Pokemon Ruby/Sapphire (gaming) ) of the American Library Association. Ms. Whitlatch has published articles and books on reference evaluation including The Role of the Academic Librarian (1990) and Evaluating Reference Services: A Practical Guide (2000).