Assessing beliefs: the epistemological scenario.
Epistemological beliefs are difficult to measure because they are covert, unconscious, and because effect of context or domain on beliefs is unclear. Current research on epistemological beliefs has primarily used three methods of data collection: Interviews, open-ended questionnaires, and Likert-scale questionnaires. This study investigated the use of scenarios as an assessment of epistemological beliefs. Results indicate that epistemological scenarios are a viable measure for assessing beliefs within a domain.
Epistemological beliefs are an individual's conception about knowledge and the conditions for acquiring knowledge (Hofer, 2002). These beliefs are challenging to measure because they are covert, unconscious, and because the effect of context or domain on beliefs is unclear (Clarebout, Elen, Luyten, & Bamps, 2001). Current research on epistemological beliefs has primarily used three types of data collection: Interviews, open-ended questionnaires, and Likert-scale questionnaires. Although all of these methods have merit, some researchers have suggested that a combination of methods within a specific context may provide a more effective means of assessment (Clarebout, et al, 2001; Hofer & Pintrich, 1997; Pajares, 1992). The purpose of this study was to investigate the use of scenarios as an assessment of epistemological beliefs that addresses these concerns.
Perry (1970) used unstructured interviews so college students could contextualize their beliefs about knowing and knowledge. The goal of these interviews was to allow students to express their beliefs without their responses being guided by a standard set of interview questions. Later studies (Baxter Magolda, 1992; Belenky et al., 1986; King & Kitchener, 1994) built on Perry's approach but began to use more structured interviews. Interview protocols and sets of open-ended questions were created, which continued to allow students to express their beliefs but focused the scope of the responses to topics such as gender, relationships, or ill-structured problems. The use of structured interviews has led to some valuable longitudinal research.
There are several problems inherent in using either structured or unstructured interviews to study epistemological beliefs. First is the issue of replicability and generalizability. Despite valuable findings in the interview research, there has been little follow-up to determine if the results can be replicated or generalized beyond the interview participants (Hofer & Pintrich, 1997). Many of these studies have been massive longitudinal investigations, which do not lend themselves to replication. Second, some interview methods require special training (e.g., the Reflective Judgement Interview; King & Kitchener, 1994), which may limit their use. Third, interview data require intensive interpretation; therefore, only small samples can be involved in research studies (Hofer & Pintrich, 1997).
The desire to study larger groups led researchers to develop more structured open-ended questionnaires. Most of these measures are based on Perry's (1970) model. For example, the Measure of Epistemological Reflection (Baxter Magolda, 1992) assesses epistemological beliefs as related to educational experiences by asking questions such as "Think about the last time you bad to make a major decision about your education (e.g., which college to attend, college major, career choice, etc.). What was the nature of the decision?" (p. 421). Student responses are then rated according to the first five of Perry's positions. Open-ended questionnaires allow students freedom of response rather than reacting to forced-choice items (Hofer & Pintrich, 1997a). However, many open-ended questionnaires, such as the Measure of Epistemological Reflection (Baxter Magolda, 1992), must also be scored by trained raters, which limit their use.
Likert-type scales have allowed data to be collected on an even larger scale. Schommer (1989) developed a 63-item Likert-scale questionnaire to assess students' beliefs. However, in such scales reliability and validity are a concern. Qian and Alvermann (1995) speculated that Schommer (1990) used principal component analysis, which has been noted by some statisticians to be an extremely liberal analysis (Cliff, 1987). Another consideration is that Likert-scale assessments, such as the Epistemological Questionnaire, may fail to provide an accurate measure of beliefs, because students are asked to respond to statements without context. For example, an item such as "Usually you can figure out difficult concepts if you eliminate all outside distractions and really concentrate" may lead to "it depends" thinking (Pajares, 1992). However, instruments, such as the Epistemological Questionnaire, allow for the study of large samples of students, which can provide insight into how these beliefs influence learning.
According to Hofer and Pintrich (1997), the existing measures of epistemological beliefs do not explain how these beliefs actually influence student learning. To gain a better understanding of how beliefs influence learning, they offered several suggestions for alternative ways of assessing beliefs. One way is to ask questions in the context of a specific domain; another is to use multiple measures to assess students' epistemological beliefs. In addition, Pajares (1992) suggested that the use of scenarios or vignettes to study individuals' beliefs would provide a richer, more accurate portrayal of beliefs.
Although scenarios, or case studies, have not been used to measure epistemological beliefs, they have been used in a variety of ways to help focus or assess student thinking. Grossman (1994), using Perry's (1970) epistemological framework, found that using scenarios was especially helpful for class discussions. Students classified as dualists by Perry's scheme were better able to contribute to class discussions when guided by scenarios rather than completely open-ended class discussions, because the scenarios provided more structure (Grossman, 1994).
Scenarios have also been used as a means to set a context for discussion in science classes (Macrina & Munro, 1995). For example, Herreid (1996) used a scenario about dissection with a large lecture introductory biology course to begin a discussion about animal rights and noted that the type of questions used to discuss the scenario also affected the scope of the discussion. The question "Do you think that students should be forced to dissect animals if they don't want to" led to a different discussion than the question "In what way can the dissection and use of animals be compared to the Nazi Holocaust?" He concluded that instructors should be aware of how questions can lead discussions when creating scenarios. Although most of the research on scenarios focuses on their value in classroom discussions, scenarios have been used as an assessment to measure students' attributions of success and failure on academic tasks (Clifford, 1986 ; Holschuh, Nist, & Olejnik, 2001). Clifford (1986) examined the effects of attributing failure to lack of effort, ability, or use of inappropriate strategies. In her study, high school students read a scenario about failure in one of three fields (business, education, or athletics) for one of three reasons (lack of effort, lack of ability, or use of inappropriate strategies). Her significant findings indicated that the scenario was an effective means of assessing students' attributions.
Overall, the research using scenarios has found that individuals are more willing to share their own views after reading about a case study, because the scenario provides a better focus for their thoughts (Echiejile, 1994; Grossman, 1994). It has also indicated that scenarios help students focus on a particular topic (Grossman, 1994). In this study, the Epistemological Scenario was used to help students think about their own epistemological beliefs within the context of their science classroom.
Participants Participants (n=518) were recruited from two large-lecture sections of Introductory Biology. The majority of participants were female (66%) and European American (69%). The class was made up of mostly freshmen (28%), sophomores (39%), and juniors (22%), which is typical in an introductory-level course. Eighty-seven percent of the participants were of traditional college age, between 18 and 21 years of age. The majority of students (89%) reported taking only one previous science course in either high school or college. For most of the students (89%), this was their first college-level science course.
Procedures Students completed the Epistemological Scenario before taking their second exam in Introductory Biology. It was believed that students would be better able to respond to the scenario after they had experienced several weeks of lecture and an exam in the course. Students were given time in class two days before their second exam to complete the assessment.
Materials The Epistemological Scenario was constructed using past research from Baxter Magolda's (1992) Measure of Epistemological Reflection, Corkhill's (1996) Metacognitive Awareness Questionnaire, and Schommer's (1989) Epistemological Questionnaire. Students read a 477-word scenario about Chris, a student struggling in introductory biology and then responded to items based on their own beliefs, even if those beliefs contradicted what was depicted in the scenario. Like Schommer's Epistemological Questionnaire, the Epistemological Scenario used a 5-point Likert-type scale with 5 standing for strongly agree and 1 standing for strongly disagree. Scores on the scenario were aggregated by item type (e.g., scores on all items about simple knowledge were summed) for analysis. All items about the scenario required students to agree or disagree with Chris's statements, an example of which is "I agree with Chris that science is based on proven facts." The scenario contained three items corresponding to each of the five epistemological dimensions hypothesized by Schommer (1990). That is, the scenario addressed the following dimensions: certain knowledge (e.g., "Like Chris, I believe that there must be one theory that is more correct than others"), innate ability (e.g.," I agree with Chris that if I don't do well on my biology exam it is because I am not able to learn science"), omniscient authority (e.g., "I agree with Chris that students should only be responsible for the scientific theories that the professor discusses in class"), quick learning (e.g., " Like Chris, I believe that I will only do well in this class if I can learn information quickly"), and simple knowledge (e.g., "I agree with Chris that learning competing science theories is confusing for students").
First, the multidimensionality of the Epistemological Scenario was analyzed as a measure of validity for the assessment. In addition, this analysis was conducted to compare the dimensions of the Epistemological Scenario with the dimensions of Schommer's Epistemological Questionnaire. The results of this analysis indicated that the Epistemological Scenario was multidimensional, and that the dimensions included certain knowledge, innate ability, quick learning, and simple knowledge.
To begin the exploration of the dimensionality of the Epistemological Scenario, an analysis of internal consistency was conducted. The results indicate Cronbach's alphas (internal consistency) ranging from .51 to .62, which are considered adequate for research purposes. This is especially encouraging for the early stages of research on this instrument (Pedhazur & Schmelkin, 1991). Second, the dimensions of the Epistemological Scenario were further investigated by exploratory factor analysis. The principal axis procedure was used with orthogonal varimax rotation. Two through five-factor solutions were tested to explore all possible models. A four-factor solution was adopted, which explained approximately 38% of the variance and had a cutoff eigenvalue of 1.23.
Results of this analysis suggested four dimensions: certain knowledge, innate ability, quick learning, and simple knowledge. Further exploration of the model was made by examining the wording of the items that loaded on the factors to determine whether the solution was justified by theoretical rationale. This analysis suggested that the four-factor model demonstrated a fairly good fit with the hypothesized dimensions and provided a theoretically sound model. The items that loaded on factor 1 fit Schommer's dimension of simple knowledge, the items that loaded on factor 2 fit Schommer's dimension of certain knowledge, the items that loaded on factor 3 fit Schommer's dimension of quick learning, and the items that loaded on factor 4 fit Schommer's dimension of innate ability. The items hypothesized to measure omniscient authority (items 2, 4, and 6) loaded on factors 1, 2, and 3. In addition, two of those items (items 2 and 6) loaded on more than one factor with a loading coefficient of greater than .30, making it impossible to determine a fit for the dimension of omniscient authority, which is consistent with previous research (e.g., Schommer, 1990, 1993b; Schommer & Walker, 1995). The results of this analysis indicated good support for the hypothesized dimensions of certain knowledge, innate ability, quick learning, and simple knowledge. Overall, the results of the factor analysis indicated that the Epistemological Scenario clearly demonstrated the dimensions hypothesized by Schommer (1990).
The analysis of the Epistemological Scenario produced satisfactory reliability of each dimension as indicated by Cronbach's alpha. The analysis produced four relatively independent factors: certain knowledge, innate ability, quick learning, and simple knowledge. The four-factor solution had a cutoff eigenvalue of 1.23 and explained 38% of the variance. Of the five dimensions hypothesized by Schornmer's (1990) model, on which the scenario was based, only omniscient authority did not appear as its own factor in the analysis. Instead, the items hypothesized to measure omniscient authority loaded onto three separate factors. Thus, the dimension of omniscient authority remains unclear. This result is consistent with previous research (Schommer, 1990, 1993b; Schommer et al., 1992), which also did not find statistical evidence for the dimension of omniscient authority.
The results of the factor analyses indicate that the Epistemological Scenario is a reliable assessment of epistemological beliefs when measuring beliefs within a specific domain. Although Schommer's (1990) model assumes that epistemological beliefs are domain independent, the effect of specific domains on epistemological beliefs is still unclear. Some research has indicated that students' beliefs are consistent across domain (Schommer & Walker, 1995), although other research has found that students' beliefs about knowledge differed depending on the domain (Hofer & Pintrich, 1997b; Jehng et al., 1993). However, no research has examined individuals' beliefs across domains. Although the results of this study do not indicate whether beliefs differ across domains, they do suggest that the domain-specific assessment provided a clear picture of an individual's beliefs.
The Epistemological Scenario may have a format that is better suited for assessing beliefs, because it provides specific context. Previous research has found that scenarios are an effective way to focus student thinking (Grossman, 1994). Research has also indicated that individuals are more willing to share their own views after reading a scenario (Echiejile, 1994; Grossman, 1994). Thus, the use of scenarios to assess epistemological beliefs may elicit a more accurate response than the more general items found on the Epistemological Questionnaire. This conclusion is consistent with Pajares's (1992) assertion that without context, students' responses to Likert-scale items any not be accurate.
Baxter Magolda, M. B. (1992). Knowing and reasoning in college: Gender-related patterns in students' intellectual development. San Francisco: Jossey-Bass.
Baxter Magolda, M. B. (1998, April). Constructing adult identities. Paper presented at the meeting of the American Educational Research Association, San Diego, CA.
Belenky, M. F., Clinchy, B. M., Goldberger, N. R., & Tarule, J. M. (1986). Women's ways of knowing: The development of self, voice, and mind. New York: Basic Books.
Clarebout, G., Elen, J., Luyten, L., & Bamps, H. (2001)Assessing epistemological beliefs: Schommer's Questionnaire revisited. Educational Research and Education, 7, 53-77.
Cliff, N. (1987). Analyzing multivariate data. San Diego, CA: Harcourt Brace Jovanovich.
Corkhill, A. J. (1996). Metacognitive awareness questionnaire scoring guidelines with person, task, and strategy components. Unpublished measure.
Echiejile, I. (1994). Training as an instrument of change: The effectiveness of case studies. Training and Management Development Methods, 8, 609-620.
Grossman, R. W. (1994). Encouraging critical thinking using the case study method and cooperative learning techniques. Journal on Excellence in College Teaching, 5, 7-20.
Herreid, C. F. (1996). Case study teaching in science: A dilemma case on "animal rights." Journal of College Science Teaching, 25,413-418.
Hofer, B. K. (2002). Personal epistemology as a psychological and educational construct. In B. K. Hofer & P. R. Pintrich (eds.) Personal Epistemology. Mahwah, N J: Earlbaum.
Hofer, B. K., & Pintrich, P. R. (1997). The development of epistemological theories: Beliefs about knowledge and knowing and their relation to learning. Review of Educational Research, 67, 88-140.
Holschuh, J. P., Nist, S. L., & Olejnik, S. (2001). Attributions to failure: The effects of effort, ability, and learning strategy use on perceptions of future goals and emotional responses. Reading Psychology, 22, 153-173.
Jehng, J. J., Johnson, S. D., & Anderson, R. C. (1993). Schooling and students' epistemological beliefs about learning. Contemporary Educational Psychology, 18, 23-35.
King, P. M., & Kitchener, K. S. (1994). Developing reflective judgement. San Francisco: Jossey-Bass.
Macrina, F. L., & Munro, C. L. (1995). The case study approach to teaching scientific integrity in nursing and the biomedical sciences. Journal of Professional Nursing, 11,40-44.
Pajares, M. F. (1992). Teachers' beliefs and educational research: Cleaning up a messy construct. Review of Educational Research, 62,307-332.
Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, N J: Erlbaum.
Perry, W. G., Jr. (1970). Forms of intellectual and ethical development in the college years: A scheme. New York: Holt, Rinehart, & Winston.
Qian, G., & Alvermann, D. (1995). Role of epistemological beliefs and learned helplessness in secondary school students' learning science concepts from text. Journal of Educational Psychology, 87, 282-292.
Schommer, M. (1989). Epistemological questionnaire. Unpublished measure.
Schommer, M. (1990). Effects of beliefs about the nature of knowledge on comprehension. Journal of Educational Psychology, 82, 498-504.
Schommer, M. (1993). Epistemological development and academic performance among secondary students. Journal of Educational Psychology, 85,406-411.
Schommer, M., & Walker, K. (1995). Are epistemological beliefs similar across domains? Journal of Educational Psychology, 87, 424-432.
Welty, W. M. (1989). Discussion method teaching: How to make it work. Change, 21 (4), 40-49.
Jodi Patrick Holschuh, University of Georgia
Holschuh, Ph.D., is an Associate Professor of Reading and Studying in the Division of Academic Enhancement
|Printer friendly Cite/link Email Feedback|
|Author:||Holschuh, Jodi Patrick|
|Publication:||Academic Exchange Quarterly|
|Date:||Jun 22, 2006|
|Previous Article:||Reducing social work students' statistics anxiety.|
|Next Article:||A problem-based learning approach to leadership.|