What a load of garbage! A data analysis exercise.

Abstract

Learning how to analyze qualitative data is difficult for beginning research students, not only because it is a long and complex process, but because it is so different from working with quantitative data, with which students are often more familiar. Working with any data requires interpretation of results by the researcher in order to reach meaningful conclusions. This article describes an exercise in qualitative data analysis that introduces university students to the processes, limits and possibilities of this challenging work.

Introduction

As a lecturer in qualitative research methods for education students at both the BA and MA levels, I am faced with the challenge of presenting qualitative research as a legitimate form of research to students schooled in a positivistic, quantitative tradition. One of the things that worries my students the most is how qualitative researchers' interpretations affect the validity of their findings.

I teach my students that there are rigorous methods of strengthening internal validity, especially triangulation of data, that is, collecting data through observations, interviews and document analysis, as well as having other researchers do reliability checks. These methods, combined with extensive time in the field and continuous reflection and self-checking by the researcher, go a long way towards helping the qualitative researcher overcome his or her biases and assumptions and arrive at conclusions that are truly data-based. (Rock, 2001, p.34).

Once the students learn these approaches they begin to see that qualitative research is not simply based on the subjective perceptions or personal interpretations of the researcher. There are techniques, methods and standards that underlie careful qualitative research. I try to explain to them as well that quantitative research is not automatically "good"--that sloppy quantitative researchers can use poorly designed questionnaires, faulty statistical designs and inappropriate samples, thus reaching unfounded conclusions. There are good and bad, careful and sloppy research studies of every kind.

After the first part of the course my students can accept that there are scientific standards for the conduct of qualitative data collection, but when we approach the subject of data analysis they have another crisis of doubt about the validity of results. There are a number of ways of approaching the analysis of qualitative data. One can do narrative analysis, looking at the structure and holistic content of stories respondents tell. One can bring categories of analysis to the research, in order to verily existing theory, even quantifying the occurrences of each category. But the classic ethnographic approach involves reading and rereading the data in order to identify patterns and themes. These become categories that the researcher uses to re-examine the data, coding items of data and refining the categories several times until a system of categories has been constructed, sensitive enough to explain all or most of the data (Fetterman, 1998, chapter 5). This is challenging, requiring patience and the tolerance for uncertainty. Most students are able to learn this kind of content analysis and to arrive at a set of descriptive categories. They start to worry again when I tell them that the descriptive level is not enough; good descriptive categories are the basis for more advanced analysis that involves the search for conceptual, rather than descriptive categories, and an interpretive leap towards a more theoretical understanding of the data. This "leap" towards theory is profoundly data-based, but employs also a kind of researcher intuition. "Intuition?" the students cry in horror. "This is what we were afraid of all along! This kind of research is based on subjectivity, gut feelings and intuition. It isn't scientific."

I try to explain that quantitative researchers also employ intuition in many phases of their work. Intuition may guide them in identifying research questions and in interpreting the significance of research findings. Polanyi (1958) writes about 'intuition' and 'insight' to describe what he called "the act of understanding" (p.91) in science and mathematics. Polanyi quotes from Einstein's autobiography, in which Einstein describes his intuitive understanding, from the time he was sixteen, of elements of the theory of relativity, later confirmed (but not discovered) experimentally (Polanyi, 1958, pp.10-11).

An exercise in data collection

I got the idea for this exercise from the following description by Merriam (1991, pp.) of how to begin qualitative data analysis:

   For a simple but vivid example of how to take raw data and sort
   them into categories, consider the task of sorting two hundred food
   items found in a grocery store ... One scheme may separate the
   items into flesh, frozen, canned or packaged goods. Or, you could
   divide them by color, weight, or price. More likely, you would
   divide the items into common grocery store categories: meat, dairy,
   produce, canned goods, and so on ... All these schemes emerge
   logically from the "data"--that is, the food items (pp. 132-133).

Approaching grocery items as data, the research question would dictate the categorization scheme chosen. There could be alternate schemes and alternate explanations of the grocery store "data" depending on the data themselves and the interpretations of the "researcher". He or she might ask, "What are these for?" or "What are their main ingredients?" or "Where are they stored in the kitchen?" Each of these questions would lead to a somewhat different categorization. This idea reminded me also of the time, while doing my undergraduate degree in anthropology, that I participated in a dig at a Northwest Coast Indian site. Excavating the remains of a 400 year old kitchen revealed all kinds of kitchen refuse which, in order to shed light on the way of life of these people, we had to sort into categories of related objects. We did this in much the same way as written data are sorted during preliminary category building. Thus the following exercise was born.

For a month or so before bringing this exercise to class I collect non-perishable items of garbage--plastic, paper, cardboard and glass packages and bottles of all sizes. My daughter, the mother of small children, also collects for me, thus introducing baby food containers, diaper packages and other items my husband I do not buy, into the mix. If I am traveling out of the country during this period I will bring back boxes and packets from potato chips, facial tissue, cookies--to add further interest and variety to the collection of "artifacts" the students will have to deal with. My goal is to have 40 to 50 items for each group. As my classes usually have about 30 students, and I like no more than five students in each group for this exercise, this means collecting at least 250 items. These are sorted into garbage bags before class, so that each group receives a garbage bag containing a similar assortment of 40 to 50 items. Every group will get egg containers, baby food jars, milk cartons, soup packages, coffee cans, items from foreign countries, etc.

This is the second class in a series of lessons related to data analysis. In the first class I introduce the basics of content analysis and the building of categories of similar "items" (expressions, sentences, events, anecdotes) of written data in order to answer the research questions. I bring examples from my own past research and describe how I have made the torturous journey, through repeated reading and rereading of the data, from preliminary descriptive categories to a more conceptual set of categories that approach the level of theory. I also explain to them the similarity between sorting archeological artifacts and written data, and the role of interpretation in explaining the meaning of the categories and how the categories are related. I refer to "interpretive glue" that binds the discrete categories into an explanatory system.

We review these ideas and then, as I tell them how great it would be if we could have an experience with archeological data, I pull the garbage bags out from under my desk and exclaim, "We just happen to have some archeological data here today!" Once the students have organized themselves into groups of five, sitting around an open space where they will dump their garbage bag for investigation and sorting, each group receives a bag and a set of written instructions. As they go through the exercise I circulate from group to group, answering questions and jotting down points for whole-group discussion at the end of the exercise. Every group receives the following scenario:

   You are archeologists in our area in the year 3000. This area has
   been unoccupied for hundreds of years, but recent excavations have
   revealed that there was a thriving civilization here until some
   natural or human disaster forced the population to abandon the city.
   Little is known of these peoples' way of life.

   The data in this bag, recovered from this site, appear to represent
   the contents of a home or neighborhood refuse site. Due to special
   conditions in the soil, these items have been preserved in
   remarkably good condition.

   Your job is to sort these items into from three to seven categories
   in order to shed light on the general research question: "What was
   the culture of these people like?" Give each category a name and be
   prepared to present your categories to the class, explaining and
   interpreting them in order to arrive at a tentative description and
   analysis of this culture.

In addition, each group receives a second research question. There are three of these; thus in the final, whole-group discussion we can compare the results from all six groups' findings about the culture, as well three pairs of groups who received identical second questions. This leads to discussion of single case study, the accumulation of information through additional cases and to what extent we can generalize from these limited findings about the culture as a whole. The additional research questions are,

   To what extent did the people in this culture have leisure time?
   What was the nature of their connections with other cultures?
   How did this culture understand issues of healthful eating?

The results of the group discussions are impressive. I have done this exercise with both Bachelor's and Master's students and have found that all approach it with interest, enthusiasm and surprising depth. The task begins with the contents of the garbage bag being dumped on the floor in the middle of a circle of five students. They sort through the items, identifying them, commenting on them, reading the labels to find out ingredients, place of origin, preparation instructions, etc. They begin a sorting process that undergoes several stages as they discuss their research questions. These discussions bring to light the complexities of categorization and interpretation and the limits and possibilities of generalization. For instance, the two groups with the question on leisure time invariably arrive at the need to define "leisure time". They see that their "data' are composed of a great many "quick" or "instant" foods (because my poor husband has to feed himself while I am off working till all hours!), and this leads them to at least two possibilities. Did the people in this culture use instant foods because they were so busy they had no time to cook (they had little leisure time), or did they do quick cooking in order to free up time for leisure activities? It becomes clear that we just don't know. We can describe these two possibilities, or variations on them, but need further research and a larger data base in order to arrive at a sound interpretation.

The two groups studying the culture's understanding of health sometimes sort data according to food group (dairy, vegetable, grain). Sometimes they sort into two basic groups, healthy food like soups and fruit juices, and less healthy foods like potato chips and ice cream. They also sometimes construct a third, "diet" category of foods among their data. In this sorting scheme the groups find that on the one hand, the people in this culture seem to love cookies and ice cream; on the other, they buy diet soft drinks and low-calorie popcorn. This leads to discussion of how to understand this finding. Perhaps some of these people ate fattening food and others ate diet foods. An alternative interpretation that groups sometimes arrive at (and this, I am always thrilled to tell them, is an interpretation that is moving in the direction of theory), is that there is a fundamental contradiction in this culture's relationship with food. The same people may have eaten both fattening and diet foods, seduced both by junk food and by the need to be thin. This is, in fact, a powerful insight into Western eating habits, and leads to some wonderful discussion about the deeper meanings of this contradiction.

In terms of general findings about the culture, the question all the groups work with, there is variety in the kinds of categories constructed and the tentative conclusions drawn. One conclusion most groups reach is that people in this culture did not eat meat (my husband and I and my daughter and her family are all vegetarians, so no products containing meat are represented). This fact is clearly represented in the data. What it means, however, is open to interpretation. Did these people embrace vegetarianism for health reasons? If so, why did they eat so much ice cream? Were there no resources for raising livestock? If not, where did their dairy products come from? Was vegetarianism a religious practice? If so, did these people perhaps eat meat in a temple or at a religious site not yet discovered? These speculations help students understand the limits of interpretation from single-case data, the ways research studies suggest further research, and how we gradually expand and deepen our knowledge through an accumulation of studies.

This exercise, which is followed up by exercises in analyzing existing written data, as well as in data collection and analysis, opens students' minds to the complexities and possibilities of working with qualitative data. Studying culture is not an easy task. Even when the participants in a culture are alive and kicking, available for interviews and willing to fill in statistically-based questionnaires, the deep structural elements of their culture remain tacit, and the researcher must make interpretive leaps in order to go beyond simple description to conceptual and theoretical analysis.

This exercise does not clear up my students' doubts about the dangers with which interpretation of qualitative data is fraught, nor should it. Rather, they begin to see the rich and fascinating potential of carefully conducted qualitative research to uncover cultural meanings, build theory and open new directions for further study.

References

Fetterman, D. (1998). Ethnography, step by step. Thousand Oaks, CA: Sage.

Merriam, S. (1991). Case study research in education. San Francisco: Jossey-Bass.

Polanyi, M. (1958). Personal knowledge. Chicago: University of Chicago Press.

Rock, P. (2001). Symbolic interactionism and ethnography. In Handbook of ethnography, eds. P Atkinson, A. Coffey, S. Delamont, J. Lofland & L. Lofland. Thousand Oaks, CA: Sage, pp. 26-38.

Deborah Court, Bar-Ilan University, Israel

Dr. Deborah Court has a doctorate in Curriculum and Instruction from the University of British Columbia in Canada. She now lives in Israel where she is a lecturer in the School Her research interests include educational cultures, teachers' practical knowledge and alternative teaching methods.