Self-Grading for Formative Assessment in Problem-Based Learning.

Abstract

Experience indicates that a major impediment to writing intensive strategies such as problem-based learning is the excessive time one must spend in formative assessment activities. A technique employed to be more efficient in formative assessment of critical thinking in problem-based learning can significantly decrease the time and work demand on the instructor, while providing timely and authentic feedback to the learner. The technique, described here for mathematics instruction, can be modified for other disciplines.

Why formative assessment?

As educators we continually ask ourselves

* How do we know our students are learning?

* How do we know they are thinking critically?

* How can we follow their patterns of thought?

But the more important questions are:

* How do they know they are learning?

* How do they know they are thinking critically?

* How can they reflect on their patterns of thought?

Background

For nearly a quarter century I taught, discussed, assigned, assessed, and graded, in some order or another. I hoped and expected that students would read and reflect on all those comments and thought-provoking questions I so meticulously wrote on their homework and tests. As I learned to give more authentic assignments, I realized that much of what we (I) had taken for learning was really patterning. If I can be allowed to oversimplify, the best student's expectation would be, "Give me an example of how to think and do, and I'll think and do like you." Students seldom evidenced their own ideas except during our Socratic questioning sessions, sessions in which formative peer- and self-assessment were part and parcel.

In the years 1991 to 1993, I had two sets of experiences that changed the way I teach. First I tried piloting a middle grades statistics book in a semester-long course for eighth graders. The only way I found to reach these students was through active learning and projects. Although some months passed before I learned to call what we were doing "problem-based learning" (PBL), the technique produced a deeper learning than even I had anticipated. After I left the middle school to return to the university, some of these students, working alone and without benefit of an advisor, produced a project which won the American Statistical Association's 1992 national project competition.

The second experience was teaching in a three-year program called Partnership for Excellence in which we modeled progressive techniques for in-service teachers. We attempted activity-based learning, guided discovery, cooperative learning--every promising technique we could find or think up; and we used technology wherever possible. What seemed to work best, as evidenced by participants' test scores and opinion polls, were the activity-based strategies.

Informed by these two experiences, some of us on the mathematics faculty at the University of South Carolina Spartanburg (USCS) fashioned a problem-based version of our College Mathematics course for liberal arts majors. We began teaching the PBL version in 1995.

The Problem

Our work began as a response to the realization that most of the students entering our math classes were and are products of pedagogical styles that depend on learning algorithms, i.e., patterning, in order to be successful (Ulmer, 1994). They bring with them no expectation that self-initiated thinking should be a characteristic of learning. There may be several reasons for this, including the following possibilities.

* Textbook dependence. One finding of the Third International Mathematics and Science Study was the inclination of U.S. grade school teachers to spend disproportionate class time using textbooks compared to such countries as Japan and Germany. Mathematics instruction in those countries was found to be more successful than in U.S. schools (Peak, 1996).

* Out-of-context instruction. Mathematics instruction has, for some decades, been governed by a curriculum in which topics were included on the basis of their consistency with other topics. Applicability, a criterion deemed important by most students, has seldom been a consideration for those writing high school and college math books. (When have you factored a quadratic polynomial, rationalized a denominator, or used the FOIL method as part of solving a real problem? Problems I encountered in my earlier work as a laboratory statistician and in more recent work as an operations research and mathematics consultant bore little resemblance to those in the textbooks from which I taught.)

* Repeated curriculum. The seventh grade textbook had the same topics as the eighth grade textbook, which was much like the ninth grade textbook.... There is the pattern again!

To encourage learner-initiated thinking, we decided to design the course without constructs that promote the patterning found in traditional courses. The problem-based version of College Mathematics at USCS uses no required textbook. Instead, a packet of activities and project assignments accompanies a forty-two-page booklet designed to add structure to the course. The booklet contains too few textbook-style problems to support the faculty member who wishes to spend class time in drill, practice, and patterning. Fewer problems are worked than in the text-based version of the course, but, as other educators using active learning and PBL have found, "students appear to emerge with a greater store of usable knowledge" (Moore, 1997). Our in-course and subsequent performance assessment data supports that statement. We have collected success-rate data from the forty-eight problem-based sections and thirty-three textbook-based sections taught through Spring 2000. With "success" defined as making a grade of D or better (so a grade off or a withdrawal is classified as "failure"), we find the following: Students succeed at a much higher rate (median = 75%) in the PBL version than in the text-based version (median = 56%) and those who take statistics the next semester succeed there at a higher rate. For more detailed data, see <http://www.uscs.edu/~mulmer/PBI_Index.shtml>.

Only about half of the mathematics faculty members at USCS have bought into the idea of PBL. Reasons have varied:

* Some do not like the prospect of abandoning texts. This is especially important since our department has several very well respected textbook authors.

* Others do not wish to, or can't, give up control of the order, methods and styles of learning to the extent necessary for PBL to work. But most of all,

* PBL presupposes teaching for critical thinking; and so, requires a great deal of formative assessment. And that means a commitment of time and effort that some faculty members cannot or will not devote to the pedagogy.

Critical Thinking and Formative Assessment

Problem-based instruction requires the learner to rely on his or her own thinking (not patterning) in order to begin solutions to problems. The problems themselves call for thinking in order to complete activities and write associated reports. This necessity for thinking allows us as instructors to teach for critical thinking, which we might loosely describe as reflective thinking, or metacognition performed for the purpose of improving the quality of thinking. Early formative assessment activities serve to acclimate the learner to this need for critical thinking.

As an example of my own development toward teaching for critical thinking and the hurdles I encountered, consider one small topic from business calculus: that of distinguishing between the Absolute Rate of Change and the Relative Rate of Change. In days gone by, I asked students to distinguish between these concepts by stating the definitions. Some students, perhaps 30%, succeeded. As I tried to improve my teaching in subsequent terms, I gave a problem where they had to use the definitions to compute absolute and relative growth of some function for a given change in the independent variable such as time. A few more, perhaps 40%, succeeded. Some years later, as benefits of contextual learning became more obvious, I actually gave the following question: "Would the Absolute Growth Rate or the Relative Growth Rate be more important to someone who wants to buy stocks?"

To my delight, half the class got it right. But that delight lasted about as long as it took you, the reader, to recognize that a higher success rate than 50% was needed on a two-response question to indicate that any learning was taking place.

Two years ago, when I got a chance to teach the course again, I began to use writing as a vehicle for promoting critical thinking. I gave the following assignment: "In one-half page or less, explain why the Absolute Growth Rate or the Relative Growth Rate (pick one) would be more important to someone who wants to buy stocks."

Realizing that for assessment to be formative, it must be timely, I graded and returned each submission with comments and tried to incorporate a consideration of common mistakes into a Socratic session. It was extremely time consuming. But even more frustrating was the realization that it was I--not the intended learners--who was thinking critically about their work. The students simply stashed my well-chosen comments in a backpack and left for history or psychology class.

The Technique

To provide opportunities for the learners to think about their work and receive timely feedback, I have used a self-grading technique comprised of the following steps:

* Give an assignment requiring a short well-thought-out written response to a question. I call it a "ticket," indicating that it must be completed for admission to the next class.

* Once in class, have students put away all books, papers, pencils, pens, etc.,--except for their responses to the assignment. Hand out colored pencils for students to use for corrections and modifications to their work.

* Solicit discussion from students about their perspectives on the assignment. Use Socratic techniques if necessary to steer the conversation. Ask all students to correct or improve their own responses as they evaluate the oral responses of their peers. Require additions or corrections to be made with the colored pencil.

* Once discussion has revealed correct or reasonable responses, provide a rubric overhead by which the student can assign his or her own grade. Take up the assignment.

* Before next class, review the students' responses and self-reported grades for accuracy and for adherence to the rubric, and make necessary adjustments in a third color of ink. Record scores.

For activity examples and its corresponding rubric, and some additional activities see the course web site given above.

Advantages

Self-grading in this manner has significant advantages over traditional instructor-graded formative assessment. Using the latter, the instructor spends time--often several hours per assignment--reading and reflecting on student responses and preparing carefully considered feedback. Ideally, the student then reads the instructor's responses, thinks critically about the comments, and prepares an improved version for the instructor to reconsider. When this process works, it requires the instructor to think critically about two submissions per assignment per student. The instructor thus hones his or her critical thinking skills far more than does the student. Impetus for thinking and learning is misplaced.

In my experience, however, instructor grading is even less efficient than this ideal. Students seldom think deeply about feedback they are given and seem quite content to address each instructor comment with a superficial correction. Thus instructor grading guarantees critical thinking only on the part of the instructor. In addition, two class periods are required for each assignment if discussion accompanies return of each draft.

The self-grading technique described above requires only one class period's discussion and appropriately directs the impetus for critical thinking. With self-grading, as with instructor grading, the instructor sees the learner's initial and second responses, but now the two responses are clearly delineated by different colors of ink and are available in only one reading. Additionally, the instructor sees that which the learner gleaned from the Socratic discussion and gets a measure of the learner's level of comprehension from her or his degree of adherence to the rubric. The time the instructor spends reviewing the submissions is significantly reduced since only small adjustments in learner understanding are generally needed.

The learner cannot avoid thinking critically in the process of self-grading. In order to formulate corrections to the initial response, the learner must reflect on the quality of his or her response and determine the degree to which it adheres to the range of possible appropriate responses generated in the Socratic session. Deciding upon and formulating corrections and improvements require additional thinking. When the graded assignment is returned from the instructor, the opportunity to think about the assignment is once again an option but so, too, is the option to stash the assignment in a backpack or portfolio. Fortunately, significant thinking has already been accomplished.

Quality and efficiency of formative assessment are both improved with self-grading as described here. The technique can be used in any disciplinary or multidisciplinary learning environment where writing-to-learn is used as a pedagogical technique. Problem-based learning, one such technique, is sometimes criticized for its propensity to allow too little content coverage. With improved efficiency derived from self-grading, concomitant improvements in content coverage of problem-based learning become realizable, and the complaint that the technique requires too much time becomes less easily supported.

References

Moore, D. S. (1997). New pedagogy and new content: the case of statistics. International Statistical Review, 65 (2), 123-137.

Peak, L. (1996). Pursuing excellence--A study of U.S. Eighth-grade mathematics and science teaching, learning, curriculum and achievement in international context (USDE Publication No. NCES 97-198). Washington, D.C.: U.S. Government Printing Office, 37.

Ulmer, M. B. (1994). "Using Statistics and Data Analysis to Motivate Other Mathematical Topics." Proceedings of the Fourth International Conference on Teaching Statistics, 2,607.

M. B. Ulmer, University of South Carolina Spartanburg

Ulmer is Professor of Mathematics and Associate Dean of the College of Arts and Sciences <[e-mail]>.

This paper is edited from the dialog of a presentation made to the Assessment 2000 conference of the American Association for Higher Education in Charlotte, NC, in June 2000.