A comparison of assessment practices and their effects on learning and motivation in a student-centered learning environment.

With recent interest in moving from teacher-directed instruction to approaches that incorporate greater opportunities for student-centered learning come concerns that traditional classroom practices associated with assessment and grading may be inappropriate for learning environments grounded in a constructivist perspective. The study reported here compared three different assessment methods used during a student-centered program. Participants were seventh graders who used Alien Rescue, a hypermedia problem-based learning program, over the course of three weeks in their science classes. The effects of the different assessment methods on students' solutions to the problem posed in the software, their performance on a factual knowledge test, and their motivational orientation were examined.

**********

Perhaps the greatest potential that computers hold to transform K-12 education lies in their ability to transform innovative educational approaches, which up until now seemed unmanageable, into viable instructional alternatives. Such is the case with student-centered learning, which, despite advocates from throughout the 20th century, failed to take hold in schools (Cuban, 1982). Yet computer-based programs designed to be student-centered must be implemented in K-12 classrooms indelibly marked by a long tradition of teacher-directed instruction. This tradition has resulted in firmly entrenched structures and procedures that are at odds with the philosophical underpinnings of student-centered learning. Among the most entrenched of these structures and procedures are those related to assessment and grading.

Student-Centered Learning

Traditionally, K-12 classes have typically been teacher-directed, meaning that teachers set objectives, determine and direct activities designed to help students meet those objectives, and assess students' learning. Teachers are the ones who normally resolve problems that their students encounter. In contrast, in student-centered environments, students are given a complex, interesting task (Jonassen, 1999), and in the process of addressing that task, they recognize the need for certain information and skills. They then identify the resources that can help them meet those learning needs. Through discussion with peers and the teacher, they reflect on their products, refining them based on the feedback they receive. In this light, it is the student who sets the objectives, directs his/her activity to meet those objectives, and assesses his/her own work. Instead of directing students' work or solving problems for students, teachers work as facilitators, helping students to think through their problems and encouraging collaboration and reflection. Theorists argue that student-centered learning makes learning meaningful (Hannafin, Land, & Oliver, 1999), supports the development of problem-solving skills, and enhances motivation (Barrows & Tamblyn, 1980). Approaches such as problem-based learning (PBL), learning by design, and project-based learning fall beneath the umbrella of student-centered learning.

Essential to student-centered approaches is student ownership of their goals and activities. Because students make decisions about, which actions to take to meet their goals, their work is meaningful to them, a condition that encourages depth of understanding and an intrinsic motivational orientation. This has been supported by our own research, in which students expressed a greater degree of intrinsic motivation for their work in Alien Rescue, a hypermedia-based PBL program, than for their normal class activities (Pedersen, 2003). However, what happens to student ownership within a student-centered learning environment when an extrinsic motivator such as grades become part of the mix? One of the reasons students in Pedersen offered for their heightened intrinsic orientation was that they did not receive grades during Alien Rescue. Therefore, one possible reason for their intrinsic motivational level may have been an absence of extrinsic motivators, such as grades.

Assessment and Grading

Assessment and grading are usually so closely linked as to be considered synonymous, but there are important distinctions between the two. Assessment focuses on decisions about both what and how aspects of students' work will be examined (e.g., performance on a test, quality of a product, observation of process through questioning, interviewing, etc.) (Stiggins, 1997). Assessment serves multiple roles; for example, it can provide feedback to learners on areas of strength or weakness and it can provide the teacher insights into the effectiveness of a given approach. How these assessments are translated into letters or numbers that quantify or summarize students' performance can be considered grading. Grading is typically the salient aspect of assessment for students, as evidenced by questions such as "Does this count?" and "What do I have to do to get an A?" that are common to discussions of school tasks.

Assessment practices that emphasize the use of grades to motivate students have long been under attack (Edwards & Edwards, 1999; Kohn, 1999), with research suggesting that these practices undermine intrinsic motivation (Butler & Nisan, 1986; Harter, 1978) and shift students' focus away from learning (Anderman & Johnston, 1998; Grolnick & Ryan, 1987). Yet, seemingly, the inevitability of grading approaches that of death and taxes. Despite three decades of research detailing the harmful effects of grades, grading persists as a common practice. Even recent efforts to implement alternative assessment methods are hampered by the concerns teachers express over the difficulties of translating largely qualitative data from verbal feedback, observations, and rubrics into letter or number grades (Seeley, 1994). Why is the practice of grading so persistent? The answer may lie in how tightly the purposes for which grading is used are woven into the tapestry of schooling. Teachers have traditionally used grades to motivate students; schools often use grades to rank or sort students; parents expect grades as a means of gauging their children's progress and effort (Kohn, 1994). A practice that serves so many roles is not one easily discarded.

Given the role that grades play in the culture of schools, teachers implementing student-centered activities must consider not only how they will assess students, but how they will translate their assessment practices into grades and justify their methods. Because of this emphasis on assessment and traditional grading structures, the feasibility of implementing student-centered approaches in traditional classes has been questioned (Land & Hannafin, 2000). Developing methods of assessment and grading that support learning and motivation in student-centered activities is essential to the success of these approaches.

Assessment Methods

Increased interest in student-centered approaches has been accompanied by calls for reforms that bring assessment methods into line with new conceptions of learning informed by a constructivist perspective (e.g., Graue, 1993; Stiggins, 1999). In her presidential address at the 2000 AERA Annual Meeting, Shepard (2000) argued that assessment methods commonly used in education today reflect the associationist and behavioral learning theories that held popular sway in the early and middle 20th century. Within that paradigm, scientific measurement of learning through objective tests was a logical practice, but as new theories of learning and instruction gained ground without a concomitant change in assessment practices, instruction and assessment became separate and ill-matched. Shepard advocated the development and investigation of new conceptions of assessment that are consonant with emerging perspectives of learning.

These new conceptions of assessment differ from traditional ones in a number of ways, two of which are pertinent to this study. First, the notion that assessment should be authentic (Wiggins, 1989) has led to a focus on how well students perform within the context of their activities. Authentic assessments, which are also called performance assessments or direct assessments, examine what students can do in terms of the types of tasks or performances for which the instruction was designed (Shepard & Bliem, 1995). In other words, the way that students perform the tasks they engage in and the artifacts they create as a natural part of their work are a better source of information about student understanding and progress than the easily gradable worksheets and objective tests students are often expected to complete, but which may seem only marginally relevant to the complex tasks posed by student-centered approaches. Second, by involving students in evaluating their own work, assessment can be seen as a tool to help students develop meta-cognitive awareness and focus student attention away from grades and toward learning goals. Research has shown that students who are involved in assessing their own work become more interested in the criteria and their own performance than in the grades they received (Klenowski, 1995), suggesting that student participation in assessment may promote intrinsic motivation. Student participation in the assessment of their work is a key aspect of what Stiggins (1997) referred to as student-centered assessment.

Purpose of the Study

Given the inevitability of grading and its potential to undermine the intrinsic motivation essential to student-centered approaches, a greater understanding of the procedures by which those grades are derived and their effects on learning and motivation is needed. Are some assessment methods better than others in supporting learning during student-centered activities? Are some less injurious or even beneficial to student motivation? The primary purpose of this study was to investigate the effects of different assessment practices as students were engaged in a student-centered learning activity. Additionally, a portion of this study is a partial replication of a previous study (Pedersen, 2003), which showed that students expressed statistically significantly higher levels of intrinsic motivation for student-centered programs such as the one used in this study than for their regular science classes. In the earlier study, the qualitative data offered potential reasons for differences in students' motivational orientations in the two settings. Among those reasons was the lack of grading during the study. Students in that study did not receive grades during the three weeks of the program, and though this was not the only reason they offered for their enhanced intrinsic orientation, it is the only condition in that study that differs from the current study. No matter what assessment method was used with their class, all students in this study received grades. Ideally, it would have been possible to design this experiment to include a control group that did not receive grades as part of the process of assessment; this would have made it possible to test the assertion from that previous study that the lack of grades contributed to students' enhanced motivation. However, the classroom teacher felt obliged to give grades during the study to all groups, so it was not possible to address this issue in this study. Therefore this study cannot draw strong inferences about the effects of grading within a student-centered activity on learning and motivation, but the replication of that previous study allows some reflection on this issue.

The research questions were

1. In a student-centered learning environment, do students perform better on tasks they know they will be graded on?

2. Do different assessment practices impact the quality of the problem solution in a PBL unit and students' acquisition of factual knowledge?

3. Do students express different levels of intrinsic motivation for their work in regular science classes and their work in programs like Alien Rescue?

4. Do different assessment methods differentially impact students' motivational orientation?

METHOD

This study used mixed methods with a quasi-experimental design. There was one independent variable (assessment method) and three treatment conditions.

Participants

Students in three 7th-grade science classes in a mid-size southwestern city participated in the study. The composition of the classes reflected that of the school at large, which reports that 73% of their students are white, 12% are African American, 9% are Hispanic, with the remainder not reported. Twenty-one percent of the students are considered economically disadvantaged. The three classes that participated in the study were all taught by the same science teacher. Classes were randomly assigned to one of the treatment conditions. Students in these classes were members of the target audience for whom the software package used in this study was designed. Eighty-two students received parental consent to participate in the study, but the scores of five students were not included in the analysis because the failed to complete all the measures used. Therefore, for this study N = 77.

Materials

Participants used Alien Rescue, a computer-based PBL program for middle school science. Students used the program for fifteen 50-minute periods over the course of three weeks.

Alien Rescue presents students with a complex problem to solve, and provides access to all the informational resources students need as well as a number of tools they can use to develop a solution to this problem. Students are placed in the role of scientists aboard a space station who are tasked with finding new homes on worlds in our solar system for each of six extraterrestrial species aboard a spaceship in orbit around Earth. These species originated in a distant solar system that was destroyed, and after sustaining damage to their ship, entered a state of suspended animation where they must stay until they arrive on their new homes. Students must learn about the aliens and the planets and large moons in our solar system so they can match each species to a new homeworld.

Among the resources available within Alien Rescue are three that are of particular importance to this study. The first is the notebook. The notebook allows students to flexibly create sections to organize their notes. Students generally record notes on the needs of the alien species, facts about worlds in our solar system that relate to these needs, and any hypotheses or questions students have that they want to remember to follow up on at a later time. The second resource is a simulation in which students can design probes to send to other worlds in our solar system to gather data that they are missing about those worlds. Designing probes requires students to deal with a number of constraints, such as choosing the correct instrument to gather the desired data, and selecting the appropriate type of probe based on which instruments are used. An inappropriate design results in malfunctions or in a failure to obtain the needed information. To aid students in planning their probes, Alien Rescue is structured so that students need to write a probe mission statement. Students are expected to record what information they plan for the probe to collect. Teachers can then use these mission statements to discuss students' designs as well as their knowledge about worlds in our solar system and science concepts related to the types of data they plan to collect. Finally, Alien Rescue provides a recommendation form where students record their solutions. This form has six sections, one for each of the species of aliens. Each section asks students to select a world in our solar system to be the new home of the species, then to provide a rationale for why that world is suitable for that species.

To support students in developing a solution for each of these species, Alien Rescue contains an expert tool. This tool offers video clips of an expert as he/she models thinking while developing a solution for one of the species. The expert reviews informational resources, records notes that are pertinent to the problem solution, designs probes to gather missing information, and writes a rationale for his choice of Mars as the new home for one of the species.

Many small problems are embedded throughout Alien Rescue, such as a need to convert between the Kelvin and Celsius temperature scales, identify substances using their spectrograms, and interpret data returned from probes. The size of the overall problem as well as the numerous small challenges students encounter makes Alien Rescue complex and students typically seek support to deal with the difficulties they encounter. However, our formative evaluation has consistently shown that students are eventually able to overcome these difficulties and articulate how they did so.

Alien Rescue was designed in accordance with the Texas Essential Knowledge and Skills (TEKS) for science. It received first place in the 2001 Learning Software Design Competition (Hooper, Hokanson, Bernhardt, & Johnson, 2002).

Treatment Conditions

This study examined one independent variable, assessment method. There were three treatment conditions, each representing a different method of assessment, and each of the three classes participating in the study was randomly assigned to one of these conditions. Students in all three treatment conditions were told that they would be graded on the solution they developed to the central problem in Alien Rescue, and that there would be a factual knowledge test at the end of the program that would also be used in determining their grade for that six weeks reporting period. Each group was then told that their work within the program would be assessed as well and that grades would be based on these assessments. All three groups received grades both during their work on Alien Rescue, and on their solutions and factual knowledge tests. The following are the three assessment methods examined in this study.

Inclass assignments. Students completed five worksheets during class time over the course of Alien Rescue. The time required to complete each of these worksheets ranged from approximately 10 to 30 minutes, though students were given an entire class period to complete each. Students were told that the worksheets would be graded by the instructor, and that their scores on these worksheets would be counted toward their grade in the course. The graded worksheets were returned to students the day after each was completed, and the teacher led a brief review of the worksheets to discuss the correct responses to each item. This assessment condition most closely resembles traditional assessment methods, in that the worksheets provided objective items that were either correct or incorrect, and that their completion could be considered a related but somewhat separate task from students' primary goal of developing a solution to the problem posed by the software program.

Artifacts. As students work within Alien Rescue, they generate a number of artifacts, including the contents of the notebook within the program, and a list and description of the probes that they design. Alien Rescue allows students to print these artifacts as well as view them within the program interface. Though all students generate these artifacts as they use Alien Rescue, only students in this group were told that they would be graded on the quality of these artifacts. This is an example of an authentic assessment method in that work that is meaningful within the context of the program is what is assessed. Students in this group also completed the same five worksheets as the above group, but their worksheets were not graded. Rather, the teacher simply returned the ungraded worksheets to students the next day and discussed their answers with them. These discussions paralleled the discussions held in the inclass assignments group.

Peer-and-self-assessment. Students in this group were assigned to teams of three or four students to work during Alien Rescue. At the end of each week, students were asked to assess their own process and progress as well as that of their teammates. To do so, the teacher worked with the students at the beginning of each week to determine a set of criteria for the assessment that would take place at the end of the week. The criteria established in this manner fell into two types: process and benchmark progress. For example, students included a score on willingness to collaborate and participate in class discussions, as well as scores on how well they had met benchmarks the class determined to be reasonable for the week. Then at the end of the week, students were given a form listing these criteria and asked to score their own work and that of their teammates on a 0 to 5 point scale. Students were given a cumulative grade based on this evaluation, but they were not told which of their teammates had assigned which scores. This assessment was conducted three times, once for each week students were engaged in Alien Rescue. Because students were assessed on aspects of their work that contributed directly to the task they were performing, and because they examined their own work and reflected on their progress, this treatment condition can be considered both authentic and student-centered. Students in this group did not complete the worksheets assigned to students in the other two classes.

Instruments

Worksheets. Alien Rescue was designed to be used in a student-centered manner, with students determining their daily activities based on a process they themselves develop to solve the problem presented by the program. Because of this, our original designs for Alien Rescue did not include worksheets for teachers to use with students. We viewed worksheets as potentially counterproductive in a student-centered learning environment because they focus students' attention on discrete tasks that may or may not be meaningful to individual students on the days in which the worksheets are assigned. However, one of the most common modifications teachers make to the program is the addition of worksheets they themselves create. The reason that teachers offer for developing and using worksheets is that they help learners to identify key information in the program, and that this in turn enhances both the quality of the solutions students develop and their learning of factual knowledge from Alien Rescue. We therefore created five worksheets to be used over the course of the program. The topics for the worksheets were determined by examining the types of knowledge students need during the program and through discussion with teachers who had developed their own worksheets. The final worksheets were reviewed by a teacher who had previously used Alien Rescue with her classes; she suggested a few minor modifications, which were implemented, and approved the content of the worksheets. These worksheets were used by two of the three classes.

Program artifacts. Two of the artifacts generated by students' work within Alien Rescue were examined for the participants in two of the three treatment conditions. Students in the artifacts condition were told they would receive grades on these two artifacts, and the grades they received were reported to them during the course of the program. The artifacts generated by students in the inclass assignments group were also examined, but students were unaware that this occurred and did not receive grades on these two artifacts. The first artifact was the contents of their notebooks, which as previously explained, is a tool contained with Alien Rescue. This artifact includes the number and names of the sections students create, as well as the contents and word count of each of the sections. To assess the quality of students' notes, the contents of the notebooks were parsed into statements, with each statement reflecting a single piece of information, such as a fact about a species, a fact about one of the worlds in our solar system, or a hypothesis about which species might be able to survive on which worlds. Each statement was then coded as relevant or irrelevant to the problem solution. Students' notebooks were scored based on the number of notes they took as well as the percent of those notes that were relevant to the solution. This score however, could not be calculated quickly enough to use it as the basis of reporting a grade to students on their notebooks. Therefore, the teacher assigned a grade based on her informal assessment of the notes, though this assessment was also based on the quantity of notes generated and the quality of the contents of those notes. The second artifact is a list of the probes students create, which includes a mission statement for each probe. Students are expected to explain their reason for launching each probe in this mission statement, and to be considered meaningful, this reason is expected to include specific pieces of information students want to gather about given worlds they are considering as potential homes for each of the alien species. Students' probe design was assessed by determining the number of probes they created and the percent of those probes for which they wrote meaningful probe mission statements.

Problem solution. In addition to the two artifacts described, Alien Rescue also records the solutions students develop for the problem, namely the world they select for each species, and a short essay in which they provide a rationale for each choice. Students are expected to develop solutions for five species. Students' solutions for each species were worth a maximum of 7 points; because there were five species for which students developed solutions, students could score a maximum of 35 points on this measure. Three criteria were used to evaluate each essay. The first was the suitability of the world chosen for the species. For each species more than one world in our solar system is viable; therefore there are multiple correct answers for each species, though some answers are better than others. Students scored two points for choosing an optimal world for a species, one point for choosing a viable but less than optimal world, and zero for any other world. The second criterion was the number of accurate supporting details included in the rationale for the choice of world. Supporting details provided specific information about the world that showed why it was suitable for a given species. These supporting details were expected to address issues pertinent to the survival of a species on the chosen world, and focused on topics such as temperature, gravity level, presence of atmosphere, composition of atmosphere and surface, presence of a magnetic field, and seismic activity. Students scored one point for 1-2 supporting details, two points for 3-4 supporting details, and three points for 5 or more supporting details. The final criterion addressed the overall quality of an essay. An essay was considered to be of high quality if the student related the qualities of a world directly to the needs of a species, discussed possible disadvantages of a chosen world, and included no inaccurate statements in the essay. Essays scored 0 to 2 points for overall quality. Two raters scored each essay. The raters scored 12 essays together in order to establish similar judgments, then scored the remaining essays individually. The results were compared. The initial inter-rater reliability was .91; the raters discussed the discrepancies until 100% agreement was achieved.

Factual knowledge test. This is a 25-item, multiple-choice test that accompanies the teacher's manual for Alien Rescue. Pilot tests have been conducted and the instrument has been revised based on these tests. Content validity of this measure has been examined through teacher review, in which four teachers who have used the program provided feedback on the appropriateness of each test item. A reliability analysis for this sample resulted in alpha = .8529.

The scale of intrinsic versus extrinsic orientation in the classroom. Students completed The Scale of Intrinsic versus Extrinsic Orientation in the Classroom (Harter, 1981) twice: once before starting Alien Rescue and a second time after completing the program. This scale consists of five subscales, which are described in Table 1. The items use a structured alternative format. Learners are presented with two alternative preferences and asked to decide which type they more closely resemble. Once they have chosen one of these preferences, they decide if that choice is "really true for me" or "sort of true for me." Items are scored on a 4-point scale, with 1 representing the maximum extrinsic orientation and 4 representing the maximum intrinsic orientation.

During the first administration, students were asked to think about their prior experiences in their regular science classes. During the second administration, students were asked to reflect on their experiences with Alien Rescue and imagine participating in similar projects in the future. These administrations were less than a month apart; therefore, differences can reasonably be attributed to the effect of the type of learning environment rather than to developmental changes.

Students' scores on this scale were used in two different ways. First, their scores on the pre and posttreatment administrations of the scale were analyzed to determine if students express different levels of intrinsic motivation for the two types of approaches (regular science classes and programs like Alien Rescue). This portion of the study is a partial replication of a previous study (Pedersen, 2003) which showed that students expressed statistically significantly higher levels of intrinsic motivation for programs like Alien Rescue than for the regular science classes. Second, students' scores on this scale were used to examine the impact of the different treatment conditions on motivation. Only students' posttreatment scores were used here because they reflect on how the different assessment methods within Alien Rescue affected students' motivation.

The reliability scores for this scale, both those included in the test manual and those found in this study, are reported in Table 1. The overall reliability for the entire scale was alpha = .85 for this sample.

Interviews. Ten students were interviewed after completing Alien Rescue and the factual knowledge test. The purpose of these interviews was to determine whether the different assessment methods affected the decisions students made about their process through the program and their opinions about how they were graded. Students were selected based on (a) their willingness to be interviewed, (b) researcher observation of their performance during Alien Rescue, with the purpose being to identify students who may have had different experiences/insights, and (c) teacher recommendation on which students might be most willing to share their insights. The interviews were semi-structured. Table 2 lists the initial questions for the interviews. Additional questions were used to encourage students to elaborate on their responses, with the primary goal being to allow students the opportunity to broach issues related to assessment and grading themselves, before these topics were introduced in the interviews. As topics related to assessment arose in the earlier interviews, additional questions were added to later interviews to check for agreement, disagreement, or other perspectives. Interviews lasted between seven and twelve minutes, and were audiotaped and later transcribed.

RESULTS

To answer the first research question, "In a student-centered learning environment, do students perform better on tasks they know they will be graded on?", students' scores on the worksheet assignments and the artifacts they created in Alien Rescue were analyzed. The scores of only two groups were included in this analysis because only two of the groups completed the worksheet assignments. Though all three groups created the artifacts included in this analysis, the artifacts created by the peer-and-self-assessment group were not included because this group may have had more time to work on these artifacts if they did not complete the worksheets. Nine dependent variables were included: one score for each of the five worksheets and four scores for artifacts students created within the Alien Rescue environment. The scores on the artifacts included the total number of notes taken, the percent of notes taken that were relevant to the problem solution, the number of probes designed, and the number of effective probe mission statements. A MANOVA was conducted with group assignment as the between subjects factor. Only cases that had scores for all nine dependent variables were included, so for this analysis N = 51.

The results of the MANOVA showed a significant difference between groups at the p < .001 level. Between-subjects effects were calculated for each dependent variable. As shown in Table 3, the inclass assignments group outperformed students in the artifacts group on four of the five worksheets, and this difference was significant on three of these worksheets. These results suggest that the students who knew they would be graded on these worksheets performed better than students who did not receive grades. The effect sizes shown in Table 3 for these three worksheets range from moderate to large (Stevens, 2002) and the mean scores for the different groups are sufficiently different that teachers would be likely to conclude that grading students on worksheets enhances their performance.

The results on the artifacts are more mixed. As shown in Table 3, students in the artifacts group wrote fewer notes and designed fewer probes than the students in the inclass assignments group. However the mean scores for the artifacts groups on the percent of notes that were relevant and the percent of probes with effective mission statements were higher than the other group's mean scores, and in one case (effective mission statements), this difference was significant. These results suggest a pattern. Students who knew they would be graded on these artifacts used them more carefully; they wrote fewer notes and designed fewer probes, but the quality of their work on each of the artifacts was better. It should be noted that the evidence here for this pattern is only moderately strong; only two of the four differences are statistically significant. However a "what if" analysis (Pedersen, in press; Thompson, 1989) on another variable, total number of notes, suggests that a statistically significant difference might have been achieved with a slightly larger sample size (N = 78).

To answer the second research question, "Do different grading and assessment practices impact the quality of the problem solution in a PBL unit and students' acquisition of factual knowledge?" students' solutions to the problem presented with Alien Rescue and their performance on the factual knowledge test were examined. The mean scores and standard deviations on these two measures are shown in Table 4. A MANOVA was conducted with these two scores as the dependent variables and group assignment as the between subjects factor. The results indicated that there was no significant difference in how the groups performed on these two measures (Wilks' lambda = .227).

To answer the third research question, "Do students express different levels of intrinsic motivation for their work in regular science classes and their work in programs like Alien Rescue?" a within subjects repeated measures MANOVA was conducted, with the five subscales of the Scale of Intrinsic versus Extrinsic Orientation in the Classroom as the dependent variables. Both group and overall means are shown in Table. 5. The results show a significant difference between the two administrations of the scale, with p = .029. Analysis of each of the subscales shows a significant difference in the curiosity and mastery subscales (p = .046 and .009 respectively), with students showing greater intrinsic motivation in the regular science classes than in programs like Alien Rescue. The effect sizes were small to moderate (eta squared = .051 and .087, respectively).

To answer the final research question, "Do different assessment methods differentially impact students' motivational orientation?" the data from both the Scale of Intrinsic versus Extrinsic Orientation in the Classroom and the interviews conducted with students were analyzed. Student responses on the scale were analyzed using a MANOVA of posttreatment scores on five subscales as dependent variables and assessment method as the between subjects factor. The results were not statistically different (p = .752), suggesting that the three assessment methods did not impact student motivation differentially.

Content analysis was conducted on the interviews in order to examine possible effects of the different assessment methods on student motivation. Data were unitized and categorized (Lincoln & Guba, 1985), with the emerging categories being either opinions of the majority of students across groups or opinions that differed between the groups.

1. Fairness. Regardless of the type of assessment condition, most students believed they were graded fairly. Only one of the students interviewed felt that the assessment method used in his class was unfair. This student was in the inclass assignment group, and said that he thought there should have been more daily grades, rather than just the five worksheets that students completed.

2. Effort. Regardless of treatment condition, students generally felt that they would have worked as hard on Alien Rescue if they had not received grades during the program. Some students responded that grades might make them work slightly harder because, as one student explained, "you want to get a good grade. It kind of puts the pressure on you." However, almost all of the students interviewed said that they would have still worked hard because they thought the program was fun and "you want to figure it out." Only one of the students interviewed said that he would have stopped working on Alien Rescue if he had been told that his work would not be graded. This was the same student who said that he felt the assessment method (inclass assignments) used in his class was unfair.

3. Additional grades. An interesting finding to emerge from the interview data was that several students were not only content to receive grades during Alien Rescue, but actually wanted them. Several students explained that they felt they had worked hard and performed well during Alien Rescue, and that additional grades would serve to raise their averages for the marking period. This suggests that students see grading as inevitable and want to be graded in circumstances where they feel they are performing well.

4. Self-assessment of performance. Students in all the treatment conditions felt that they had done well in Alien Rescue, though they differed slightly in the evidence they offered to support their opinions. The students in the self and peer assessment group did not reference grades as a measure of their success. Instead, they described their successes and failures in tasks such as launching probes, choosing an appropriate world for each species, and developing well-reasoned rationales for their solutions. The data from the other two treatment groups differed in that several of the students interviewed either referred to grades as their means of assessing their performance or referred to what they had done "wrong" based on grades they had received.

5. Active learning. The opportunity for active learning in both Alien Rescue and their regular science classes positively impacted students' attitudes toward learning. Almost all the students said that they enjoyed Alien Rescue and thought this was a better way to learn science than simply reading a textbook. However, several students pointed out that in their regular science class they engaged in many hands-on activities, and in a few cases, students argued that they preferred their regular science class to Alien Rescue because of this.

6. Assessment-related concerns. Almost all of the students interviewed were content with the way their work had been assessed, though a few concerns were raised. Students in the self and peer assessment group all said that they had felt some apprehension about grading themselves and their peers, but in the end all thought it worked well. Two of these students said they would like for assessment to be handled the same way on other projects, though one student said she preferred for the teacher to grade assignments. One student in the inclass assignments group explained that he was concerned with the amount of time the worksheets took, and was concerned about being able to finish the program in the allotted time. He felt that the worksheets had some value, but saw them as a distraction from the real work of solving the problem posed by Alien Rescue. Other students felt that the worksheets were useful, despite the time they required.

DISCUSSION

Assessment Methods

The results of this study fail to show the superiority of one assessment method over another in this student-centered learning environment. There were no significant differences between groups in the quality of the solutions they developed to the problem posed in Alien Rescue, the factual knowledge test, or the posttreatment administration of the Scale of Intrinsic versus Extrinsic Orientation in the Classroom. Problem-solving, factual knowledge acquisition, and motivation all appeared to be unaffected by the type of assessment used. The qualitative data indicate a few minor differences between the groups in terms of the impact of these different assessment methods. For example, students in the peer-and-self-assessment group expressed some mild anxiety about the responsibility of grading themselves and peers, but by the end of the program this had passed.

The qualitative data results suggest a possible explanation for the lack of a difference between groups based on assessment method used. First, students in all the treatment conditions enjoyed Alien Rescue. They described it as fun and most of them said that they would have worked hard on it even without grades. This means that much of the effort they invested may not have been a result of "working for a grade," which may have attenuated the motivational aspect of assessment and grading. Second, students generally felt they had been successful in the program and that they had been graded fairly, meaning that these issues never became ones that caused them anxiety. Given those conditions, it is possible that students simply saw assessment as a typical aspect of their school work, and that the details of how that assessment was conducted were unimportant to them. Had any of the groups received poor grades or perceived the assessment method to be unfair, the issue might have been a more important one and might have impacted their motivation. Or, had Alien Rescue been less enjoyable, students who did not receive grades for inclass assignments or program artifacts might have invested less effort than students who did, and their learning might have suffered.

Another factor that may have influenced these outcomes is the duration of the study. Neither the classroom teacher nor the students involved in this study had had any experience with student-centered assessment practices, so the peer-and-self-assessment group may not have had enough opportunity to consider and apply assessment methods that were novel to them. Though previous research has shown that grading can undermine performance even in extremely short treatments, the differences between assessment methods (all of which lead to grades) may be subtler and accrue only over longer periods of time. In particular, student-centered assessment methods, in which students participate in setting goals and assessing their progress toward them, may require a longer period of time and greater amount of discussion and practice with students before their effects can been seen.

One finding from the qualitative data related to the impact of different assessment methods does merit additional investigation. Students in the inclass assignments and artifacts groups were more likely than students in the peer-and-self-assessment group to cite grades when describing the quality of their performance, though the difference was not pronounced. Still, the finding that students in the self-and-peer-assessment group relied more heavily on the primary evidence taken from their performance rather than the secondary evidence available through grades to assess their work does suggest that student involvement in assessment may help to maintain student focus on the task rather than the grade. However, the evidence in this study is too weak to support such a claim, though it does provide a warrant for further investigation. Such work should be conducted within a variety of student-centered learning environments to determine if student-driven assessment better supports student focus on their process and the quality of their products as a measure of success than do more traditional teacher-directed assessment practices.

Grading

Is grading a valuable activity in student-centered learning? Grading, which has been used extensively to enhance student motivation by providing extrinsic reasons for students to invest greater effort in their learning, arguably has little value in student-centered learning environments, where student exert ownership over their work, and are therefore, by definition, intrinsically motivated. This study cannot answer that question, but two findings are relevant to it. First, and unsurprisingly, the results of this study suggest that students perform better on tasks when they know they will be graded on them. However, the results also suggest that the value of grading these tasks is questionable. Students in the peer-and-self-assessment group performed as well on both the problem solutions and factual knowledge test as students in the other conditions, yet did not receive grades on either inclass assignments or the artifacts they generated while using the program. In other words, though students in two of the groups invested greater effort in tasks on which they knew they would be graded than students in the third group who were not graded on these tasks, this investment ultimately did not result in better performance or learning. These results are consistent with two different possible explanations. The first is that grading assignments within a PBL activity has no impact on learning; that is, even though grades may motivate students to perform better on individual assignments, grading ultimately does not enhance either the quality of students' products or learning. This explanation would suggest that grading is of little value within student-centered learning environments and that assessment of student learning can be divorced from traditional grading practices without negatively impacting learning. The second possible explanation of these results is that it does not matter what products or performances are assessed in a PBL activity as long as students receive grades for something. The particulars of the assessment practices used are irrelevant to learning, but the expectation of receiving grades for some aspect of their performance motivates students to work and achieve. Unfortunately it was not possible in this study to use a true control group that did not receive grades because the classroom teacher felt compelled to provide grades to all students. It is therefore impossible to reflect upon which of these two possible explanations is more viable. However, the first explanation, namely that grading is unnecessary in a SCLE, is consistent with the results of a study conducted by Moeller and Reschke (1993). They found that grading middle school students' oral practice activities did not result in higher oral proficiency, and suggested that when students perceive activities to be "hands-on, interactive, and relevant, the students will perform the task to the best of their ability regardless of 'grading'" (p. 169).

Second, the results of the motivation scale (The Scale of Intrinsic versus Extrinsic Orientation in the Classroom) contradict those of a previous study (Pedersen, 2003), in which students expressed a greater intrinsic orientation for programs like Alien Rescue than for their regular class activities. In the current study, students expressed a significantly greater intrinsic motivational orientation for their regular science classes on two of the subscales in the motivation scale. A number of factors may have affected this. For example, the science teacher whose students participated in this study is widely considered to be excellent, and she regularly engages students in a variety of stimulating, hands-on activities. Students may not have seen Alien Rescue as a huge departure from the types of activities they regularly engage in, and therefore did not express more intrinsically motivated behavior. Though this and other explanations are plausible, the results raise an important question. A key difference between this study and Pedersen is that students in this study received grades and those in the previous one did not. One reason offered by students in Pedersen for their enhanced motivation was that they were not graded for the three-week duration of the program. This freed them to focus on what they were interested in. Again, because there was no control group that did not receive grades, it is impossible to infer the cause of the results seen here, but the discrepancies between the two studies indicate a need for a greater understanding of the role of grading in student-centered learning environments.

Using Assignments to Structure Student Work in a SCLE

The results also call into question the value of using worksheets to structure students' work during a student-centered activity. The use of these worksheets was predicated on our observations of teachers' practices in using Alien Rescue and their reasons for wanting to assign worksheets. Neither of the groups who completed these worksheets outperformed the group who did not complete them. Though a plausible explanation for the failure of these worksheets to benefit learning is that they were poorly designed or addressed inappropriate content, it should be noted that their design was informed by the practices of teachers who had previously used Alien Rescue, and that a teacher had been consulted and had agreed to their validity. Also, the qualitative data suggests that students found these worksheets to be relevant. The failure of one set of assignments to benefit learning within one student-centered learning environment is insufficient basis upon which to inform the design of other student-centered activities, but it does raise an issue that merits further investigation, namely whether teachers should attempt to structure students' work within a student-centered learning environment by providing assignments designed to focus their attention on key information.

CONCLUSION

As student-centered activities become more commonplace in schools, they will either need to be adapted to the realities of current school norms, or educators will need to work to change those norms to make schools more conducive to these approaches. The results of this study offer some encouragement that norms regarding assessment and grading, which may be considered to be counter-indicated for student-centered learning, may in fact not be terribly problematic. In a stimulating environment, assessment may be of little importance to students. They take for granted that grading, the aspect of assessment that is most salient to them, is inevitable. They will do what is necessary to make good grades, as evidenced by students' enhanced performance in this study on tasks for which they knew they would be graded. But the method by which those grades are derived has little impact on them, at least, as was the case in this study, when they do not perceive that process to be unfair or harsh. If this is the case, then even teachers who are unwilling to part with familiar assessment methods grounded in traditional teacher-directed approaches may be able to successfully implement these student-centered approaches in their classrooms.

Table 1 Subscales in the Scale of Intrinsic versus Extrinsic Orientation
in the Classroom

Subscale   Description                          Reliability  Reliability
                                                reported in  in this
                                                test manual  study

Challenge  Does the student like hard,
           challenging work or prefer easier
           assignments?                         .78 to .84   .85
Curiosity  Does the student work to satisfy
           his or her own interest, or to
           satisfy the teacher and obtain good
           grades?                              .68 to .82   .57
Mastery    Does the child prefer to do his/her
           own work and figure out problems on
           his or her own, or does the child
           rely on the teacher for help and
           guidance in these areas?             .70 to .78   .65
Judgment   Does the child feel he/she is
           capable of making judgments about
           what to do, or does he or she
           depend on the teacher's opinion
           about what to do?                    .72 to .81   .64
Criteria   Does the child know when he/she has
           succeeded on school assignments or
           tests or is the child dependent
           upon external sources of
           evaluation, such as teacher
           feedback or grades?                  .75 to .83   .70

Table 2 Initial Interview Questions

Do you think you learned much about the solar system during Alien
Rescue?
Do you think it's a good way to learn about science?
How would you compare it to learning about science in your regular
classes?
Would you want to do other programs like this?
How do you think you did in Alien Rescue?
Were you satisfied with your grades in Alien Rescue?
Do you think that you were graded fairly?
What do you think of doing those worksheets?
Did they help you, or were they in the way?
Would you have worked as hard on this program if you had known that you
wouldn't be getting grades on it?
If the teacher had said, "Okay, for the next 3 weeks, no grades" would
you still have worked on the program?
Do you have any other recommendations for people who design programs
like this?

Table 3 Scores on Inclass Assignments and Program Artifacts

Dependent             Mean and standard deviation     p     Partial eta
measure                                                     squared
               Inclass assignments  Artifacts group
                       group

Worksheet 1        85.5 (15.2)         75.0 (12.1)    .013  .120
Worksheet 2        83.1 (5.0)          76.0 (5.5)    <.001  .248
Worksheet 3        62.4 (19.7)         68.9 (15.3)    .564
Worksheet 4        91.8 (9.3)          84.0 (11.1)    .005  .148
Worksheet 5        89.0 (16.2)         81.5 (19.5)    .201
Total number
of notes           95.2 (59.2)         62.9 (33.0)    .113
Percent of
notes that
were relevant      85.8 (17.6)         90.1 (8.2)     .810
Number of
probes
designed           10.6 (4.5)           6.8 (2.9)     .031  .091
Percent of
probes with
effective
mission
statements         52.9 (32.2)         81.4 (27.0)    .002  .187

Table 4 Scores on Problem Solutions and Factual Knowledge Test

Dependent measure       Inclass      Artifacts   Peer-and-self-
                        assignments  group       assessment
                        group                    group

Problem Solutions       24.8 (5.1)   21.6 (8.5)  25.0 (6.9)
Factual Knowledge Test  19.1 (4.9)   16.4 (5.3)  19.0 (5.3)

Table 5 Pre and Posttreatment Means and Standard Deviations by Group on
the Scale of Intrinsic versus Extrinsic Orientation in the Classroom

Subscale      Inclass         Artifacts       Peer-and
              assignments     group           -self-
              group                           assesment
                                              group
            Pre      Post   Pre     Post    Pre     Post

Challenge   2.73     2.63   2.36    2.48    2.47    2.36
            (.61)    (.83)  (.66)   (.43)   (.43)   (.62)
Curiosity   2.58     2.48   2.39    2.49    2.46    2.43
            (.50)    (.56)  (.45)   (.48)   (.45)   (.53)
Mastery     2.86     2.58   2.76    2.60    2.60    2.55
            (.48)    (.60)  (.47)   (.54)   (.61)   (.54)
Judgment    2.97     2.84   2.71    2.87    2.96    2.65
            (.47)    (.51)  (.53)   (.56)   (.44)   (.53)
Criteria    2.82     2.81   2.63    2.68    2.66    2.59
            (.56)    (.58)  (.52)   (.60)   (.54)   (.45)
Total      13.96    13.33  12.85   12.93   13.14   12.77
           (1.80)   (2.11) (1.74)  (2.06)  (1.72)  (1.51)

Subscale    All Groups
            Pre     Post

Challenge   2.52    2.49
            (.59)   (.72)
Curiosity   2.48    2.43
            (.47)   (.52)
Mastery     2.75    2.58
            (.48)   (.57)
Judgment    2.88    2.79
            (.49)   (.54)
Criteria    2.70    2.67
            (.54)   (.55)
Total      13.33   13.01
           (1.70)  (1.89)

References

Anderman, E.M., & Johnston, J. (1998). Television news in the classroom: What are adolescents learning? Journal of Adolescent Research, 13, 73-100.

Barrows, H.S., & Tamblyn, R.M. (1980). Problem-based learning: An approach to medical education. New York: Springer 

Butler, R., & Nisan, M. (1986). Effects of no feedback, task-related comments, and grades on intrinsic motivation and performance. Journal of Educational Psychology, 78(3), 210-216.

Cuban, L. (1982). Persistence of the inevitable: The teacher centered classroom. Education and Urban Society, 15(1), 26-41.

Edwards, C.H., & Edwards, L. (1999). Let's end the grading game. Clearing House, 72(5), 260-263.

Graue, M.E. (1993). Integrating theory and practice through instructional assessment. Educational Assessment, 1(4), 283-309.

Grolnick, W.S., & Ryan, R.M. (1987). Autonomy in children's learning: An experimental and individual difference investigation. Journal of Personality and Social Psychology, 52(5), 890-898.

Hannafin, M., Land, S.M., & Oliver, K. (1999). Open learning environments: Foundations, methods, and models. In C. M. Reigeluth (Ed.), Instructional design theories and models: Vol. 2. (pp. 115-140). Mahwah, NJ: Lawrence Erlbaum.

Harter, S. (1978). Pleasure derived from challenge and the effects of receiving grades on children's difficulty level choices. Child Development, 49, 788-799.

Harter, S. (1981). A new self-report scale of intrinsic versus extrinsic orientation in the classroom: Motivational and informational components. Developmental Psychology, 17(3), 300-312.

Hooper, S., Hokanson, B., Bernhardt, P., & Johnson, M. (2002). A learning software design competition. Educational Technology, 42, 5-10.

Jonassen, D.H. (1999). Designing constructivist learning environments. In C. M. Reigeluth (Ed.), Instructional design theories and models: Vol. 2. (pp. 215-239). Mahwah, NJ: Lawrence Erlbaum.

Klenowski, V. (1995). Student self-evaluation processes in student-centered teaching and learning contexts of Australia and England. Assessment in Education: Principles, Policy, and Practice, 2(2), 145-163.

Kohn, A. (1994, October). Grading: The issue is not how but why. Educational Leadership, 38-41.

Kohn, A. (1999). From degrading to de-grading. High School Magazine, 6(5), 38-43.

Land, S.M., & Hannafin, M.J. (2000). Student-centered learning environments. In D. H. Jonassen & S. M. Land (Eds.), Theoretical foundations of learning environments (pp. 1-23). Mahwah, NJ: Lawrence Erlbaum.

Lincoln, Y., & Guba, E. (1985). Naturalistic inquiry. Newbury Park, California: Sage Publications.

Moeller, A. J., & Reschke, C. (1993). A second look at grading and classroom performance: Report of a research study. The Modern Language Journal, 77, 163-169.

Pedersen, S. (2003). Motivational orientation in a problem-based learning environment. Journal of Interactive Learning Research, 14(1), 51-77.

Pedersen, S. (in press). Effect sizes and "what if" analyses as supplements to statistical significance tests. Journal of Early Intervention.

Seeley, M.M. (1994, October). The mismatch between assessment and grading. Educational Leadership, 4-6.

Shepard, L.A. (2000). The role of assessment in a learning culture. Educational Researcher, 29(7), 4-14.

Shepard, L.A., & Bliem, C.L. (1995, November). Parents' thinking about standardized tests and performance assessments. Educational Researcher, 24, 25-32.

Stevens, J.P. (2002). Applied multivariate statistics for the social sciences (4th ed.). Mahwah, NJ: Lawrence Erlbaum.

Stiggins, R.J. (1997). Student-centered classroom assessment. Upper Saddle River, NJ: Merrill.

Stiggins, R.J. (1999). Assessment, student confidence, and school success. Phi Delta Kappan. 81(3), 191-198.

Thompson, B. (1989). Asking "what if" questions about significance tests. Measurement and Evaluation in Counseling and Development, 61, 334-349.

Wiggins, G. (1989). A true test: Toward more authentic and equitable assessment. Phi Delta Kappan, 70, 703-713.

SUSAN PEDERSEN 

Texas A & M University USA

[e-mail] 

DOUG WILLIAMS 

University of Louisiana at Lafayette USA