The critical incident technique and the qualitative evaluation of the connecting libraries and schools project.
This article describes Flanagan's Critical Incident Technique (CIT) for those seeking guidance in its application. Examples in the library and information science (LIS) field are discussed, including an indepth example of a CIT study conducted as part of a qualitative evaluation of the Connecting Libraries and Schools Project (CLASP) in New York City. The CLASP study analyzed critical incidents from 2,416 fifth and seventh grade students regarding their perceptions of interactions with urban public librarians and library staff. For both positive and negative critical incidents, the most important factor in these preadolescent's perception of successful library visits is the attitude of the librarian or staff member they encounter.
This article discusses Flanagan's (1954) Critical Incident Technique (CIT) and describes its applications in the library and information science (LIS) field. The CIT is a qualitative method designed to draw out the most memorable aspects of an event or experience from the study's participants (Ruben 1993). It has been used to evaluate programs or services and to inform their improvement. CIT questions typically have this format: Remember a time when you had a successful (specify activity)? Please describe. What was it about (specified activity) that made it successful? Or the negative: Remember a time when you had an unsuccessful (specify activity)?
Guidance is provided by this article for those considering using CIT in their research by discussing an example of a CIT study conducted by the author as part of a qualitative evaluation of the Connecting Libraries and Schools Project (CLASP). CLASP, a citywide program of The New York Public Library, the Brooklyn Public Library, and the Queens Borough Public Library, was evaluated by collecting critical incidents from questionnaires completed by 2,416 fifth and seventh grade students to gather their perceptions of encounters with public librarians.
USING THE CRITICAL INCIDENT TECHNIQUE IN A LIBRARY SETTING
CLASP was created by The New York Public Library in 1991 through a grant from the DeWitt Wallace-Reader's Digest Fund and was expanded to all five Manhattan boroughs in 1994 (Tice, 2001). The author was asked to conduct a qualitative evaluation of the impact of CLASP on students and chose the CIT as the most suitable method. Gathering students' critical incidents allowed analysis to center on how effective CLASP had been in changing or improving student's attitudes toward the public library. (1) The study had a twofold purpose: (1) evaluating CLASP, and (2) advancing knowledge of preadolescent perceptions of librarian/library staff encounters in the urban public library setting. (2) The CIT was selected as an appropriate method for achieving these purposes.
Flanagan's (1954) seminal article on the CIT documents the technique's origins and provides guidelines for its use. According to Flanagan, the CIT has five key stages, each of which will be discussed below with examples of its application in the CLASP evaluation:
1. General aims
2. Plans and specifications
3. Collecting the data
4. Analyzing the data
5. Interpreting and reporting
Stage 1: General Aims
During this stage, the key decision is to determine the purpose of the study: What does the researcher want to find out (Redman, Lambrecht, & Stitt-Gohdes, 2000)? Also, the researcher needs to conduct a literature review to discover what is already known about the type of activity, program, or participants to be studied. The CIT is useful for evaluating particular activities such as conducting outcomes assessments (Bycio & Allen, 2004; Jacoby & O'Brien, 2005) and measuring customer satisfaction (Arnold, Reynolds, & Ponder, 2005). Specifically, the researcher using CIT is looking to find out "precisely what it is necessary to do and not to do if the activity is to be judged successful or effective" (Fisher & Oulton, 1999, p. 113). In the case of the CLASP project, the study's purpose was program evaluation.
Stage 2: Plans and Specifications
Before any data can be collected, sufficient preparation is necessary. One important decision is identifying the subjects from whom critical incidents will be collected. In the case of the CLASP study, fifth and seventh graders, aged nine to twelve years old, were targeted because they would have experienced several CLASP classroom visits and they would have the writing skills necessary to complete the questionnaires. This highlights another decision to be made--the method of data collection. Critical incidents can be collected by observation, face-to-face (individual or group) interviews, telephone interviews, or paper or email questionnaires. This decision will be influenced by budgetary and staff considerations. In the CLASP study, project librarians were available to administer the paper questionnaires at the various school sites, to distribute and collect the surveys, and to assist students as needed.
Data collection through observation or face-to-face, group, or telephone interviews is much more labor-intensive when compared to paper or email survey distribution, but it has the advantage of providing the ability to ask probing or clarification questions of the subjects. This leads into the next decision point--determining who will collect the data. For a local project, such as the CLASP review, existing staff can be used. It is possible for a single researcher to collect critical incidents, even for multiple research sites (see Ozkaramanli, 2005; Radford, 1993, 1996, 1999). For a project with a larger, perhaps national, scope, it is necessary to gather a team of researchers (Redman, Lambrecht, & Stitt-Gohdes, 2000).
It is important to develop a plan for recruiting subjects. In the public or academic library setting it may be necessary to over-recruit and to offer an incentive such as the $15.00 value card issued to student interview participants reported by Jacoby and O'Brien (2005). Even with this incentive, only five of twelve students recruited actually showed up for the interview (Jacoby & O'Brien, 2005). Group interviewees are frequently recruited with offers of refreshments. Budget considerations usually dictate the range of incentives that can be offered.
Stage 3: Collecting the Data
For all data collection methods, a data collection instrument (survey or interview schedule) must be constructed and training materials or instructions for the data collection team must be developed. It is critically important that those collecting the data be thoroughly trained. Redman, Lambrecht, and Stitt-Gohdes discuss specifics of conducting the CIT interview, which they recommend as a "powerful tool" for data collection (2000, p. 136). For group or individual interviews, it is highly recommend that they be audio-taped with permission of the participants (see Radford, 1993, 1999).
In determining what questions to ask, it is useful to visualize the final report or paper to be written and to think about what questions will help to gather the information needed for that report. In commercial marketing, CIT questions could be as simple as "What did you like most (least) about your airline flight today? Why?" For the CLASP project, each student was asked to recall and describe in their own words: (a) a successful library experience either recently or in the past (that is, a positive critical incident), (b) an unsuccessful library experience (that is, a negative critical incident), and (c) the factors that made the experience successful or unsuccessful. Flanagan (1954) reported that participants provide 10 percent more incidents if asked about positive incidents first. Additionally, appropriate demographic questions should be included in the survey instrument to help in describing the subjects. The CLASP questionnaire included demographic questions as to the participant's gender, grade level, and languages spoken at home (see Appendix A for CLASP questionnaire).
Decisions on when to collect the data and how much data to collect must also be made. "Data collection may take place while the activity is ongoing, e.g., by supervisors, or by reports of fairly recent activity" (Fisher & Oulton, 1999, p. 114). In the case of CLASE the data was collected after a series of school visits, gathering perceptions and recollections of the student subjects. Because of the nature of CIT research, large numbers of incidents are frequently collected, but this varies widely. For example, Hamer (2003) collected and analyzed eight CIT interviews of approximately seventy-five minutes each. On the other end of the spectrum, research reported here analyzed 2,416 critical incidents for the CLASP program evaluation. In deciding on sample size it is important to realize that the amount of time required to analyze large amounts of qualitative data whether collected by questionnaires or interviews requires significant commitment of budgetary and human resources.
Flanagan (1954) provided some guidance on sample size, indicating that it must be determined based on the type of activity to be studied. "The underlying rationale is not to be able to make statistical generalizations but rather to ensure that the whole content domain is covered" (Fisher & Oulton, 1999, p. 114). Decisions about how many incidents to collect are unique to each project and depend on such factors as available budgetary and staff resources, purpose and intended use of critical incident results, and target audience for findings. One strategy is to decide upon a range (for example, 50 to 100) of incidents, to collect the minimum number, to begin analysis, and to see if content categories are saturated. If new categories are continuing to emerge with each incident, continue data collection. If not, stop.
The CLASP sample was drawn from all participating schools (private and public) throughout New York City. It was selected in consultation with representatives from The New York Public Library, the Brooklyn Public Library, and the Queens Borough Public Library and took into consideration which schools would be willing to participate and facilitate distribution of the questionnaires. A minimum of one class of fifth and/or seventh graders was chosen from each of the 68 selected schools, thus determining the resulting number of 2,416 students in the sample. All CLASP staff who assisted in distribution of the questionnaires were given written instructions (see Appendix B for CLASP instructions) and, in March of 2000, all questionnaires were administered.
Stage 4: Analyzing the Data
By far the stage that requires the largest investment in time is stage four. The goal of analysis for any research project is to make sense of a large mass of data through data reduction techniques that summarize and describe the data efficiently (Fisher & Oulton, 1999). For qualitative data, content analysis is frequently performed to identify common themes within the data (see Hamer, 2003; Radford, 1993, 1999).
During stage four the data is transcribed (if interviews were audio-taped) or typed into a word processor or software package. Data should be organized into files or notebooks with each critical incident being coded with a unique number. Incidents are then carefully read and sorted into content themes in an iterative process. Much has been written on the process of analyzing qualitative data (for example, see Strauss & Corbin, 1998). Content analysis and theme development is a subjective process, requiring that more than one researcher be involved in data analysis. Researchers generally use a second or third rater to look at theme development for a portion of the data (10 percent to 20 percent or more). Having a high level of inter-rater agreement adds reliability to the findings (see Radford, 1999; Redman, Lambrecht, & Stitt-Gohdes, 2000).
One way to organize the analysis is to sort the data by research question. In the CLASP study, the theoretical foundation of Watzlawick, Beavin, and Jackson (1967) provided a framework to begin sorting the data by "relational" or "content" dimensions (see Radford 1993, 1996, 1999 for a description of this process of developing themes using the Watzlawick, Beavin & Jackson framework). Software packages are available to assist in this process. (3) Data can also be analyzed by hand or by using a spreadsheet (see Redman, Lambrecht, & Stitt-Gohdes  for discussion of use of a code sheet for hand or spreadsheet analysis). One technique for manual sorting of large amounts of textual data is the Multichromatic Analysis Technique (MAT), which "involves use of colored markers and colored paper clips as aids to analysis" for large qualitative data sets (Radford, 1999, pp. 46-47; see Radford, 1993 or Skiba-King, 1993 for a description of the MAT developed by de Chesnay).
Using the MAT, CLASP student responses to the critical incident questions were sorted into categories. Responses to the survey questions to provide "good" and "bad or unpleasant" experiences were sorted into large preliminary categories. Responses were further grouped into nine content (information) and four relational (affect) themes following the work of Watzlawick, Beavin, and Jackson (1967), Goffman (1959) and Radford (1993, 1996, 1999). Content-oriented statements tended to focus on information exchange while relational-oriented statements dealt with feelings and attitudes of the participants, which were indicative of the type of relationship between the librarian/staff member and preadolescent user.
Stage 5: Interpreting and Reporting
It is important to document method and decision points along the way. If this is done in a consistent manner, the report will be easier to write. The audience should be kept in mind when constructing the research or project report. A two-page executive summary should be provided as an introduction to a report written for administrators or funding agencies. When writing for publication, detailed method and procedures descriptions are extremely important. For all audiences, the results and interpretation (discussion section) will be of high interest, so emphasize these. For interpretation of results, a conceptual framework is recommended. "Interpretation of the findings is dependent upon a solid grounding in a conceptual base that allows conclusions and educational implications to be drawn from the thematic base" (Redman, Lambrecht, & Stitt-Gohdes, 2000). A return to the stage one review of the appropriate literature, or an expanded review, may be fruitful in suggesting a framework for interpreting data.
The findings should be presented in a form that is useful to the intended audience and should be accompanied by an evaluation of the limitations, validity, and reliability of the method (Fisher & Oulton, 1999). When reporting results, numerical counts of theme frequency can be provided, or alternatively, themes can be listed in order of frequency but without the actual counts. Representative quotations from participants for each theme should be identified and included in the report to provide interesting illustrations and to help the reader understand and contextualize the findings. The report below of the CLASP results gives the numerical counts of theme frequency and includes representative quotations.
RESULTS Of CLASP CIT ANALYSIS
A total of 2,452 questionnaires were distributed and returned for a return rate of 100 percent. Of these, 36 were judged unusable (illegible or all questions left blank). There were 2,416 usable questionnaires for analysis (98 percent), (4) an unusually high number for survey research. The high return rate and number of usable questionnaires is thought to be due to the method of distribution in which the CLASP librarians handed out and collected the questionnaires in each selected class. Overall, students were remarkably forthcoming, and their written responses represent their feelings and beliefs quite eloquently, as seen in their quotations below.
Questionnaires were received from a citywide total of 68 schools in 20 districts. There were 382 usable questionnaires from Manhattan, 557 from the Bronx, 93 from Staten Island, 218 from Brooklyn, and 1,166 from Queens. Of the usable questionnaires, 1,077 (45 percent) were from boys and 1,291 (53 percent) were from girls; 48 (2 percent) did not indicate gender. There were 1,270 (53 percent) fifth grade students and 1,146 (47 percent) seventh grade students. The students were from diverse ethnic backgrounds, seen in the variety of languages spoken at home. English alone was spoken in 428 homes, Spanish alone in 68 homes, and both Spanish and English in 309 homes. A total of 60 additional languages and language combinations were spoken in 1,591 homes.
Positive Critical Incidents--Content Themes
What do students remember as being critical to their having had a successful visit to the library? A total of 1,680 students responded to the positive CIT question that asked them to think about times they had a good experience in the public library and to write down what happened and what they thought it was that made this visit good. Students reported that they had positive experiences in the library when they found the books or information they were looking for, used the computers, attended library programs, and the library atmosphere was pleasant. The content- (information) related responses were grouped into nine themes: Found Good Book, Used Computer/Internet, Found Information for School Assignment, Attended Library Programs, Got Library Card, Enjoyed Atmosphere/Facilities, Found Good Videos, Learned Information/Skills, and Found Different Formats. These are listed below, with representative student quotes as examples. (5)
Found Good Book(s) (468)
When I went to the library after school. I was doing my homework. I saw a good book. I was so delighted, I borrowed it. This was a good visit to the library because I felt like I was inside the book. (fifth grade boy)
Used Computer/Internet (327)
Being able to go on the internet. Don't have computer at home so I'm not able to go on the internet very often. (seventh grade boy)
Found Information for School Assignment (for example, Reports, Projects) (250)
I needed to find information so I could finish my report and they had everything I needed. I can always count on the library to get the right information for me. (seventh grade girl)
Attended Library Programs (133)
One time when I was little I went to the library and I sat down and listened to a story where the person who said it did origami at the same time. It was a good story and me and my brother liked it a lot. (fifth grade girl)
Got Library Card (98)
When I got my library card and I found out I could take out up to 10 books. (seventh grade girl)
Enjoyed Atmosphere/Facilities (74)
When I first went to the library I loved it because it was quiet and peaceful and the librarians let me go on the computers and I made an essay and report on black history. (fifth grade, gender not indicated)
Found Good Video(s) (54)
[It was a good experience in the library] when I got my adult card. Also when I started to take out videos. I had never done this before, so it was very exciting. (seventh grade boy)
Learned New Information/Skills (49)
One time I needed an article from the New York Times and they gave me a roll of a newspaper on a film. It made this a good visit when they taught me how to use it on a machine. (seventh grade girl)
Found Different Formats (16)
It was the first time I went to a library. I was so happy because I found a lot of cool books, CDs, movies, and computers. I got a 100 on this small project and it was fun doing my work in the library. (seventh grade boy)
Positive Critical Incidents--Relational Themes
Students also reported remembering good visits when they were treated well by librarians, felt good about themselves, enjoyed the company of friends and family, and had a variety of positive experiences. These relational responses were sorted into four relational (affect) themes: Librarian Attitude, Social Aspects, Positive Experiences/Emotions, and Felt Good about Self. They are illustrated with representative examples below.
Librarian/Staff Attitude (275)
When a guy in the [branch] library helped me find information on my research paper. Even though there were a lot of people he helped everyone. He even helped me look at the passage to see if there was any information I needed. It made a good visit because [before] when I went to the library to ask for help they just wrote a number and [said to] look it up for yourself. (seventh grade girl)
Social Aspects (188)
Once I went to the library, and I met my friend there. We researched for our project. We asked the librarian and she helped us. [What made it a good visit was] that I went to the library with my friend & we both helped each other. After that we went out to eat. (seventh grade girl)
Positive Experiences/Emotions (108)
When I was picking out a book on money I found real money in the book. [It was good] because if I didn't go to the library I would've never found the money. (fifth grade boy)
Felt Good about Self (50)
Someone was looking for a book and they didn't know where it was so I said "you need some help" and I got the book for him. I felt [like a] grown woman and I felt like a librarian. (fifth grade girl)
Negative Critical Incidents--Content Themes
What do students remember as being critical in determining unpleasant experiences in the library? In order to find the answer to this question, students were asked to think about times they had visited the public library and to remember a time when they had an unpleasant experience in the library. They were asked to write down what happened and what they thought it was that made this an unpleasant visit. Nine hundred and eighty-four student responses to this question were analyzed and sorted into five content (information) and four relational (affect, emotional) themes. It is important to note that 17 percent of the 984 students responded that they never had a bad experience in the library.
Students reported that they had unpleasant experiences in the library when they could not find the books or information they were looking for; had problems at checkout (usually related to their having overdue books or fines) ; had problems using or getting access to the computers; or the library atmosphere was problematic (usually noisy or crowded). The content- (information) related responses were grouped into five themes--Could Not Find Information, Checkout/Procedural Problems, Computer Problems, Library Atmosphere Problematic, and Negative Experiences--and are illustrated with representative examples below.
Could Not Find Information (150)
A time it had not been pleasant is when I went to the library to look for a book and it is not there. I do not feel good when that happens. What made this a bad visit to the library was they didn't have the book that I needed and I felt that all libraries should have everything you need. (fifth grade girl)
Checkout/Procedural Problems (94)
I remember now it was when they looked at my file and found out I had a list of overdue books and I had to pay a big fine. They found out that I had a list of overdue books waiting to be paid. At first I was nervous and scared because I thought they would just get mad at me and make me pay right away. (fifth grade boy)
I took out a book and they told me that there were some books I didn't return. The lady at the front was rude. She kept on telling me that I owed stuff I never took out. (seventh grade girl)
Computer Problems (68)
When I went to the library I couldn't find any books on my project. So, I signed on the computer, but I had to wait around 2 hours. At home my printer was broken, and I had to wait 2 hours. The librarian was mean too. (seventh grade girl)
Library Atmosphere Problematic (47) Once when I went to the library everybody was talking loud and was yelling. The visit was unpleasant because I went there to study and everybody was talking so I couldn't concentrate. (seventh grade boy)
Negative Experiences (30)
The printer took nay money. (fifth grade girl) My parents forced me to go. I couldn't play that day. I was missing all the fun. I could have been playing. (seventh grade boy)
Negative Critical Incidents--Relational Themes
Students reported that they had negative or unpleasant experiences in the library related to how they were treated by the librarians/staff; when they encountered other users who were problematic; when they got in trouble, were reprimanded, or were asked to leave ("kicked out") ; or when they were embarrassed or afraid. The relational responses were grouped into six themes: Librarian/Staff Attitude Poor, I Got "Kicked Out," Other Users Problematic, I Got in Trouble, Negative Emotions, and I Was Embarrassed; they are illustrated with representative examples below.
Librarian/Staff Attitude (152)
Oh yeah, the first time I went there was a very mean lady there. She was very bad and fox nothing yelled at everyone and on purpose. She kicked 5 people out. And I never went back to nay library. (seventh grade girl) The librarian kicked me and my friends out, but we weren't the ones talking. The librarian was so rude and was accusing the innocent. (seventh grade girl)
I Got "Kicked Out" (71)
When I was talking then they said "be quiet" and I was quiet, but they kicked me out anyway. [It was a bad or unpleasant visit because] I was not talking but they kicked me out anyway. (fifth grade boy)
Other Users Problematic (59)
A bad experience was when some teenagers came to me and were very rude and obnoxious. The teenagers were out of control and the librarians didn't stop or say anything to them. (seventh grade girl)
I Was Embarrassed (42)
I felt embarrassed when I tripped and fell and everybody saw me. (seventh grade boy)
I Got in Trouble (25)
I got in trouble because I stayed on the computer 5 minutes too long. The librarian kept on screaming because I was eating gum. (fifth grade girl)
Negative Feelings (19)
A bad time in the library was when I could not read. I just got a book and was just looking at the pictures. I did not know how to read or spell. (fifth grade boy)
THE CIT: DISCUSSION AND LIMITATIONS
The above report of results with representative quotations shows the richness of the results from the CIT method and how it can be used for program evaluation and to formulate specific recommendations for program improvement. Based on the CIT evaluation of CLASP, numerous recommendations were developed to enhance and foster positive student experiences in the library and to assist students in having fewer negative experiences.
One of the most interesting findings from this research is that the technique of having fifth and seventh grade urban students write critical incidents is such a productive method of data collection. (6) Since the population is so diverse, with so many languages represented, one might expect that the students would not be as articulate as is demonstrated here. In general, fifth graders were found to be more forthcoming in their answers and wrote longer answers. This finding may be due to developmental differences in which the younger students are not as reluctant as the older ones to reveal their emotions. Interestingly, boys were just as likely as girls to be forthcoming and to give answers related to relational as well as content topics.
Despite some reluctance, which is to be expected in preadolescent populations, students revealed themselves to be quite sensitive. They can be easily embarrassed, have a keen sense of fairness, and resent injustice, especially when they feel that they have been falsely accused. They report vivid memories of times when they were "yelled at," "scolded," or asked to leave and write that they were, at times, deeply affected by their fears and insecurities. For example, one seventh grade girl who was asked to leave the library during her first visit wrote: "And I never went back to my library." She felt that she had been unfairly reprimanded by a librarian or staff member, so she left and chose never to return (see also Radford & Radford, 1997).
Another noteworthy finding is that for both positive and negative critical incidents, the largest numbers of responses for the relational category were centered on "Librarian/Staff Attitude." Thus, for these preadolescents, the most important factor in their perception of successful library visits is the attitude of the librarian or staff member they encounter. They appreciate it when the librarian/staff member takes an interest in them, shows flexibility in bending the rules or makes exceptions for them (for example, extends computer time for their assignments), and accompanies them to find a book on a shelf instead of pointing or providing a call number on a slip of paper. Radford (1993, 1996, 1999) reported similar findings regarding college-age academic library users. Thomas notes that "in some cultures the relational 'work' to establish a basis for further interaction must precede information tasks" (1999, p. 159), which is affirmed by the data analysis above. Students need to be reassured that they will have a positive interaction before they feel comfortable enough to approach the librarian/staff person or reveal their information need (see also Kuhlthau, 2004). Furthermore, Bialo and Sivin-Kachala (1996) assert that relational, interpersonal aspects of library encounters are especially important for school librarians serving culturally and ethnically diverse school populations.
The application of communication theory to interpersonal encounters of preadolescents in the library context is at the exploratory stage. One criticism of library literature has been its lack of robust theory. The CIT provides a method that enables the application of the heuristically rich relational theory from the communication field to library interactions. In addition, collecting critical incidents elicits the user's point of view to advance knowledge of the preadolescent perspective in library interactions.
The results of this CIT analysis argue for a new model of the librarian-preadolescent reference interaction that takes a process approach and recognizes the vital importance of the interpersonal, relational messages that are communicated in the encounter along with the transfer of information, instructions, or suggestions for research strategies (see also Kuhlthau, 2004). It also integrates the user's perspective as critical to understanding this complex encounter (Morris, 1994). Related research in virtual reference (chat) encounters (such as Ask-A-Librarian services) has found that interpersonal aspects similar to those of face-to-face interactions are present, and they are extremely important to success, especially in interactions with adolescents and preadolescents (see Radford, 2006a, 2006b; Radford & Thompson, 2004).
Regarding practical implications, this study suggests that education for children's and young adult librarians could benefit from increased study of interpersonal dynamics. According to Winston and Paone:
Young adults currently represent a large and important segment of the user population in public libraries and population estimates indicate that this group will be a growing segment as well. However, the research literature and practice-oriented literature in library and information science have not addressed a number of issues associated with the service provision and characteristics of this user population. (2001, p. 49)
If librarians were made more aware of the impact of their actions on the fragile egos of preadolescents, they might be more cautious in administering discipline, striving not to "accuse the innocent," or inadvertently driving students away from the library or discouraging them from asking questions. Furthermore, cultivating positive relationships with preadolescents and adolescents will multiply the pleasant interactions and minimize the negative ones. Winston and Paone (2001) found that public libraries did not give priority to services to the young adult population and that it is necessary to maintain sufficient staffing levels, especially age-level specialists, to meet their needs. Whether or not public service librarians are age-level specialists, it is possible to improve the quality of encounters with youthful clients. Although many library practitioners believe that "people skills" are inherent, research findings demonstrate that librarians can be educated to improve their interpersonal skills in reference encounters (Dewdney, 1987).
This research is exploratory, and, although a large number of students were surveyed, no claim for generalization of these results is made. Limitations of the CIT research design include the fact that data are self-reported, in this case by students, and, as such, are subjective accounts of their perceptions. In addition, the schools surveyed were not selected randomly but were chosen in recognition of access and facilitation issues. Surveys were administered and collected by CLASP librarians, which may have impeded some students from answering candidly.
LIS Research Using the CIT
According to Fisher and Oulton, "The Critical Incident Technique has been tried and tested in a wide range of discipline areas and for a variety of purposes. It is recognized as a valid, reliable and effective method for gathering rich qualitative data for a variety of purposes" (1999, p. 126). The CIT has been successfully used in many social science fields, in marketing and business applications, and in the LIS context (see Andersson & Nilsson, 1964; Andrews, 1991; Carr, 1980; Fivars, 1980; Radford, 1999; Shirey, 1991). A brief overview of some of the projects in which the CIT is used in LIS is provided here to illustrate the variety of these applications.
Fisher and Oulton's (1999) article provides an excellent overview of the CIT and also describes three studies by researchers in the UK that apply the technique to research: staff development needs in the context of change in UK higher education, decision-making practice in European libraries, and developing a tool to support library workers entering management positions.
Hamer (2003) used the CIT to investigate information seeking of young gay males regarding coming out and took a social constructionist perspective of gay identity. CITs were collected in audio-taped interviews lasting approximately seventy-five minutes with eight volunteers in their late teens or early twenties. Results indicate that subjects had three types of information needs: self-labeling, consequences for self-identifying as gay, and forming an understanding of gay identity. In addition, Hamer found that their information seeking was characterized by the experience of fear and by the need for concealment and secrecy.
In her doctoral dissertation, Ozkaramanli (2005) used the CIT to study the perceptions of librarians of quality digital reference services. Critical incidents were collected through interviews with forty librarians from ten academic libraries in Ohio and Pennsylvania that offered chat reference services. Findings revealed that librarian and user attitudes, question negotiation, and availability of resources were critical to perceived success in chat interactions. Ozkaramanli (2005) provides a detailed explication of the CIT and her data analysis technique.
Radford (1993, 1996, 1999) used the CIT to study interpersonal communication aspects of reference service in academic libraries, collecting forty-seven critical incidents from twenty-seven academic librarians and twenty-seven library users. Radford's work was based on the theoretical foundation of Watzlawick, Beavin, and Jackson (1967) and Goffman (1959) that differentiated the dual nature of messages as having both content and relational dimensions. Watzlawick, Beavin, and Jackson's (1967) perspective draws attention to the idea that more than correct answers to reference questions are being communicated in reference encounters. In addition to providing this content information in response to a user's request, librarians are also communicating relational information in their verbal and nonverbal expressions that has additional meaning for library users.
The CIT is a flexible tool able to be applied in a variety of settings and for a variety of purposes. As can be seen in the above results and discussion, the CIT has provided a method for tapping into student's perceptions, yielding a substantial data set for program evaluation and a qualitative analysis of the communication process. Qualitative measures, such as the CIT, help to capture the differences that may fall between points on a standard scale. They can answer such questions as "What do programs mean to participants? What is the quality of their experience?" (Patton, 1987, p. 30). They capture nuances of quality that are lost in most survey data collection. Clearly, the CLASP study has only begun to explore the dynamics of the complex interaction between librarians, library staff, and young people. With increased understanding of this process, greater success and satisfaction for both preadolescent users and librarians can be achieved. The CIT provides a method for expanding one's understanding of this interaction, especially from the young person's point of view. It is hoped that this article will stimulate interest in the CIT and will encourage LIS researchers to consider adopting this method in future projects, including program evaluation.
APPENDIX A CLASP QUESTIONNAIRE Connecting Libraries and Schools Project (CLASP) STUDENT QUESTIONNAIRE
DIRECTIONS: Please write your answers to the questions below. Try to answer all questions. Use your best handwriting or printing. Do not put your name on this form; all answers will be kept private. Your answers are very important to us. Thanks!
Gender (circle one): Girl Boy
Grade Level (circle one): 5th 7th
Languages (circle all languages spoken at home) :
English Spanish Chinese Russian Creole Urdu Farsi Other--
1. Think about times you have visited the public library. Remember a time when you had a good experience in the library. Please write down what happened.
1A. What was it that made this a good visit to the library?
2. Now remember a time when you had an unpleasant or bad experience in the library. Please write down what happened.
2A. What was it that made this a bad or unpleasant visit to the library?
3. Can you remember a time when a public librarian came to your classroom to talk about the library? Circle one: YES NO
3A. If yes, have the public librarian visits changed your experiences in the public library? Circle one: YES NO
3B. If yes, how have the librarian visits changed your experiences in the public library?
4. Is there anything else you would like to tell us about the public library?
Thank you for helping us to make your public library better for you and for other students.
APPENDIX B CLASP INSTRUCTIONS
Instructions for CLASP Staff
Student Questionnaire Collection
Here are instructions for the person administering the questionnaire:
1. Give 1 copy of the questionnaire to each student.
2. Tell the students: "This is not a test. There are no right or wrong answers."
3. Tell the students: "If you answer these questions, it will help us make the library better for you."
4. Tell the students: "Do not put your name on the paper. Your answers will be private and confidential."
5. Read each question to the students, starting with School. Make sure the students answer every question. If necessary, help them to understand what the question means. Give them about 4 or 5 minutes for each essay answer.
6. Collect all forms at the end of the class period. Send the forms immediately to EP/CLASP. Do not wait until you have visited all the schools. Complete the collection of all forms before March 31, 2000.
Andersson, B., & Nilsson, S. (1964). Studies in the reliability and validity of the critical incident technique. Journal of Applied Psychology, 48, 398-403.
Andrews, J. (1991). The use of the critical incident research technique in an academic library. Library & Information Research News, 14(50), 22-27.
Arnold, M.J., Reynolds, K. E., & Ponder, N. (2005). Customer delight in a retail context: Investigating delightful and terrible shopping experiences. Journal of Business Research, 58(8), 1132-1145.
Bialo, E. R., & Sivin-Kachala, J. (1996). The effectiveness of technology in schools: A summary of recent research. School Library Media Quarterly, 25(Fall), 51-57.
Bycio, P., & Allen, J. S. (2004). A critical incidents approach to outcomes assessment. Journal of Education for Business, 80(2), 86-92.
Carr, D. (1980). The agent and the learner: Interactions in assisted adult library learning. Public Library Quarterly, 2(2), 3-19.
Chelton, M. K. (1997). The "overdue kid:" A face-to-face library service encounter as ritual interaction. Library & Information Science Research, 19(4), 387-399.
Chelton, M. K. (1999). Structural and theoretical constraints on reference service in a high school library media center. RUSQ, 38(3), 275-282.
Dewdney, P. (1987). User satisfaction with the public library reference interview: An experiment to determine the effects of training librarians in communication skills. Unpublished doctoral dissertation, School of Library and Information Service, University of Western Ontario.
Fisher, S., & Oulton, T. (1999). The critical incident technique in library and information management research. Education for Information, 17(2), 113-126.
Fivars, G. (1980). The critical incident technique: A bibliography (2nd ed.). Palo Alto, CA: American Institutes for Research in the Behavioral Sciences. (ERIC Document Reproduction Service No. ED 195 681).
Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 5, 327-358.
Goffman, E. (1959). The presentation of self in everyday life. Garden City, NY: Doubleday Anchor.
Hamer, J. S. (2003). Coming-out: Gay males' information seeking. School Libraries Worldwide, 9(2), 73-79.
Jacoby, J., & O'Brien, N. R (2005). Assessing the impact of reference services provided to undergraduate students. College & Research Libraries, 66(4), 324-340.
Kuhlthau, C. C. (2004). Seeking meaning: A process approach to library and information services (2nd ed.) Westport, CN: Libraries Unlimited.
Mellon, C. A. (1995). "She's nice and she helps:" What 6th graders say about school librarians. School Library, Journal, 41(5), 27-29.
Miles, M. B., & Huberman, A. M. (1994). An expanded sourcebook: Qualitative data analysis (2nd ed.). Thousand Oaks, CA: Sage.
Morris, R. (1994). Toward a user-centered information service. Journal of the American ,Society for Information Science, 45(1), 20-30.
Ozkaramanli, E. (2005). Librarians' perceptions of quality distal reference services by means of critical incidents. Unpublished doctoral dissertation, University of Pittsburgh.
Patton, M. Q. (1987). How to use qualitative methods in evaluation. Newbury Park, CA: Sage.
Radford, M. L. (1993). Relational aspects of reference interactions: A qualitative investigation of the perceptions of users and librarians in the academic library. Unpublished doctoral dissertation, Rutgers, The State University of New Jersey.
Radford, M. L. (1996). Communication theory applied to the reference encounter: An analysis of critical incidents. Library Quarterly, 66(2), 123-137.
Radford, M. L. (1999). The reference encounter: Interpersonal communication in the academic library. Chicago: ACRL, a Division of the American Library Association.
Radford, M. L. (2000). CLASP--Connecting Libraries and Schools Project, 1999-2000. Unpublished report for The New York Public Library, Brooklyn Public Library, and Queens Borough Public Library, May 23.
Radford, M. L. (2006a). Encountering virtual users: A qualitative investigation of interpersonal communication in chat reference. Journal of the American Society for Information Science and Technology, 57(8), 1046-1059.
Radford, M. L. (2006b). Interpersonal communication in chat reference: Encounters with rude and impatient users. In R. D. Lankes, E. Abels, M. White, & S. N. Haque, (Eds.), The virtual reference desk: Creating a reference future (pp. 41-73). New York: Neal-Schuman.
Radford, M. L., & Radford, G. P. (1997). Power, knowledge, and fear: Feminism, Foucault and the stereotype of the female librarian. Library Quarterly, 67(3), 250-266.
Radford, M. L., & Thompson, J. (2004). "Yo Dude! Y R U Typin So Slow ?" Online proceedings of the Virtual Reference Desk 6th Annual Conference, Cincinnati, OH, November 8-9. Retrieved May 21, 2006, from http://www.webjunction.org/do/DisplayContent?id=12 499.
Redman, D. H., Lambrecht, J.J., & Stitt-Gohdes, W. L. (2000). The critical incident technique: A tool for qualitative research. Delta Pi Epsilon Journal, 42(3), 132-53.
Ruben, B. D. (1993). What patients remember: A content analysis of critical incidents in health care. Health Communication, 5, l-16.
Shirey, D. (1991). Critical incident technique. In A. Kent, (Ed.), Encyclopedia of Library and Information Science (Vol. 6, pp. 286-291). New York: Dekker.
Skiba-King, E. (1993). An examination of the patterns of self-reported disclosure by incest survivors. Unpublished doctoral dissertation, Rutgers, The State University of New Jersey.
Strauss, A., & Corbin, J. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory (2nd ed.). Thousand Oaks, CA: Sage.
Thomas, N. P. (1999). Information literacy and information skills instruction: Applying research to practice in the school library media center. Englewood, CO: Libraries Unlimited.
Tice, M. (2001). Queens Borough Public Library and the Connecting Libraries and Schools Project. Journal of Youth Services in Libraries, 14(3), 11-13.
Watzlawick, E, Beavin, J., & Jackson, D. (1967). Pragmatics of human communication. New York: Norton.
Winston, M., & Paone, K. L. (2001). Reference and information services tot young adults: A research study of public libraries in New Jersey. Reference & User Services Quarterly, 41(1) : 45-50.
The author would like to thank The New York Public Library, Brooklyn Public Library, and Queens Borough Public Library for permission to publish these findings. The author also extends heartfelt thanks to Kate Todd, Grace Shanrahan, and Margaret Tice for their guidance and assistance and to all the CLASP librarians and students who took part in the evaluation project.
(1.) The CLASP evaluation included survey questionnaires about the CLASP program, including questions designed to document the impact of the CLASP visits on student's perceptions of the public library. Other than the CIT questions, these results are not reported here in detail (Radford, 2000; Tice, 2001).
(2.) The CLASP evaluation included survey questionnaires about the CLASP classroom visits and other aspects of the program. As noted above, other than the CIT questions, these results are not reported here in detail. However, it should be noted that the results were very positive, with 75 percent of students surveyed remembering CLASP visits and large numbers of students reporting positive impacts of CLASP (Radford, 2000; Tice, 2001).
(3.) There are software packages available to assist in analysis, such as NVivo, MAXqda, and Atlas.ti, but such packages require a substantial investment of funds and time to learn how to use them effectively. See Miles and Huberman (1994) for a description of this type of software product.
(4.) All percentages are rounded to the nearest whole number.
(5.) For all quotations throughout this paper, the number of responses is given in parenthesis and themes are listed in descending order of frequency. Note that student answers can be sorted into more than one theme when more than one concept is present. Students' minor errors in spelling or grammar have been corrected and words have been added [in brackets] as necessary for clarification.
(6.) It is to be noted that research involving preadolescents' and adolescents' perceptions of librarians has been scarce, although some researchers have investigated this area from a qualitative perspective in the school library context (for example, Chelton, 1999, 1997; Mellon, 1995) and through survey research in public libraries (for example, Winston & Paone, 2001).
Marie L. Radford is an Associate Professor at Rutgers SCILS. Prior to joining the Rutgers faculty, she was Acting Dean and Associate Professor of Pratt Institute's School of Information and Library Science. She holds a Ph.D. from Rutgers University and an M.S.L.S. from Syracuse University. Before arriving at Pratt in 1996, she was Head of Curriculum Materials at William Paterson University of New Jersey's Cheng Library and a school librarian in Hunterdon Count, NJ. Her research interests are interpersonal communication aspects of reference service (traditional and virtual), evaluation of digital resources and services, qualitative methodologies, cultural studies, and media stereotypes of librarians. She has given numerous international, national, and local presentations at scholarly library and communication conferences. She has published in Library Quarterly, College & Research Libraries, Library Trends, the Journal of Education for Library & Information Science, JASIST, and the Journal of Academic Librarianship. Her latest book, Web Research: Selection, Evaluation, and Citing (2nd ed.) was published by Allyn and Bacon in 2006. Her book The Reference Encounter: Interpersonal Communication in the Academic Library was published by ACRL/ALA in 1999. She is active in library and communication associations, including the ALA. NJLA, ALISE, and RUSA. She is the past chair of the Library Research Round Table and presently a member of the LRRT nominating committee. She is presently a member of the RUSA Publications Committee and the ALA Research & Statistics Committee. She co-chaired the ALISE conference program in 2004 and is a member of the program committee for the upcoming Library Research Seminar IV.