Improving classroom assessment practices: a collaborative approach.

Abstract

The purpose of this article is to discuss the assessment component of the Competency and Confidence Project (C^2). C^2 was a collaborative effort between the Colorado School of Mines (CSM) and the Jefferson County School District (JCS). CSM provided the project experts in mathematics content and instructional pedagogy; JCS provided expert mathematics teachers. These individuals worked together to create the C^2 workshops.

**********

Teachers are being asked to select or create classroom assessment tasks and scoring rubrics that are consistent with district, state and national standards. Researchers have argued that in order for teachers to meet these requests, they will need additional training in the assessment process (Brookhart, 1999; Stiggins, 1999). Stiggins (1999) has further argued that the focus of traditional teacher training is inadequate in that it emphasizes the interpretation of standardized tests rather than classroom assessment efforts.

The purpose of this article is to discuss the assessment component of the Competency and Confidence Project (C^2). C^2, which was partially supported by Colorado Eisenhower funds, was a collaborative effort between the Colorado School of Mines (CSM) and Jefferson County School District (JCS). Both CSM and JCS are located near Denver, Colorado. CSM offered expertise in mathematics content and instructional pedagogy; JCS provided expert mathematics teachers. These individuals worked together to create the C^2 workshops. A portion of this project focused directly upon assisting teachers in developing assessment tasks and scoring instruments that were consistent with the district's algebra standard, "Students use algebraic methods to explore, model, and describe patterns and functions involving numbers, shapes, data, and graphs in problem-solving situations and communicate with appropriate mathematical language and reasoning used in solving these problems" (Jefferson County Public Schools, 1996, p. 10).

Participating Teachers

In response to a printed solicitation, twenty-eight fourth through sixth grade mathematics teachers enrolled in C^2 in the fall of 1998. All of the teachers were from JCS. The experience level of the teachers ranged from 1-28 years, with a median of 11 years.

Objectives of the Assessment Component of Project

C^2 consisted of a series of workshops that were designed to improve teachers' content and pedagogical knowledge with respect to the mathematics that they taught. A major emphasis of the pedagogical workshops was classroom assessment. Four eight-hour, pedagogical workshops were offered throughout the semester. The primary objectives of the assessment component of these workshops are listed below.

Pedagogical Workshop 1:

1. The teachers will be able to create assessment tasks that are clearly aligned with the algebra standard.

Pedagogical Workshop 2:

2. The teachers will learn to create rubrics in which the scoring criteria are consistent with the information that the assessment task is likely to elicit from students.

3. The teachers will develop rubrics in which the performance levels are clearly distinguishable.

4. The teachers will provide to their students feedback that can be used to improve future performances.

Pedagogical Workshop 3:

5. The teachers will learn the meaning of the terms "validity" and "reliability" and these concepts will be used to reinforce the information discussed in previous workshops.

Pedagogical Workshop 4:

6. The teachers will be aware of the standardized tests that will be administered throughout the academic year.

Submitted Materials

During each of the four workshops, three hours was set aside for teamwork. These teams developed a lesson plan, an assessment activity and a rubric that was aligned with the algebra standard. Each teacher fine tuned the resultant materials and implemented these materials in the classroom. Copies of the teacher's final assessment task, scoring rubric and a sample set of scored student responses were submitted to the project evaluator. Each teacher submitted four separate sets of classroom assessment materials. The sequence of activities was as follows: 1) workshop (team activity), 2) classroom implementation, and 3) submission of assessment materials. This was repeated four times throughout the semester.

Scoring Rubrics

The submitted materials were evaluated using the Classroom Assessment Materials Scoring Rubric (CAMS rubric). The CAMS rubric is an analytic rubric with four facets: "Standard", "Criteria", "Language Specificity" and "Feedback to Students". Ms. Jill Fellman who has extensive experience and training in the classroom assessment process and is an experienced classroom teacher developed the original version of the rubric. Dr. Moskal, an assessment specialist, reviewed and assisted in the revision of the final rubric. A description of the different facets is provided below.

Standard

Level 4: The proposed task elicits evidence of students' knowledge of the algebra standard.

Level 3: The proposed task elicits evidence of mathematical knowledge and skills that directly support the attainment of the algebra standard, but does not directly address the algebra standard.

Level 2: The mathematics assessed in the task is at a basic skill level.

Level 1: The task does not have a clear mathematical foundation or the mathematics is incorrect.

Criteria

Level 4: The established criteria reflect the mathematical content assessed in the task and clearly indicates how student work will indicate essential characteristics.

Level 3: The criteria reflect the mathematical content assessed in the task, but in some cases does not clearly indicate how students work will display these characteristics.

Level 2: Some of the criteria clearly reflect the mathematical content assessed in the task, but also non-content related displays are required.

Level 1: The criteria do not reflect the mathematical content assessed in the task.

Language Specificity

Level 4: The language that is used is specific and clearly discriminates between score levels.

Level 3: Subjective language is used to discriminate between some of the score categories.

Level 2: Subjective language is used throughout to discriminate between the score categories.

Level 1: Vague language is used to describe the score categories.

Feedback to Students

Level 4: Feedback provided to students is aligned with the rubric. Students have a clear indication as to how to improve their performances.

Level 3: Feedback provided to the students is aligned with the rubric. Students have some indication as to how to improve.

Level 2: Feedback provided to the students is consistent with the rubric.

Level 1: No feedback is provided to the students.

The "Standard" category examines the extent to which the proposed assessment task is likely to elicit evidence of students' knowledge that was directly related to the algebra standard (workshop objective 1). The "Criteria" category examines the extent to which the requirements that were set forth in the task are aligned with the requirements that were set forth in the scoring rubric (workshop objective 2). The "Language Specificity" category examines the extent to which the language that is used in the teachers' submitted rubrics clearly distinguishes each of the score categories (workshop objective 3). The "Feedback to Students" category examines the extent of feedback that the teacher provided on the student papers (workshop objective 4).

Two objectives are not directly addressed through the CAMS rubric, workshop objectives 5 and 6. Objective 5 concerns introducing the terms "validity" and "reliability" with the purpose of reinforcing earlier objectives. Therefore, objective 5 is indirectly assessed through the other score categories. Objective 6 concerns standardized testing and is not the focus of this paper.

The "Total Score" for each submission was calculated by summing across the teachers' scores with respect to the given categories. This resulted in each teacher receiving four "Total Scores", one for each submitted set of assessment materials. The "Total Score" was designed to capture changes in the teachers overall classroom assessment performance from the first to the fourth submission. The use of total scores is consistent with prior recommendations concerning scoring rubrics (Mertler, 2001).

Results

Reliability

Two raters scored five common sets of submitted materials. They then compared their scores and through discussion, clarified the categories. Next, the two raters independently scored 20% of the submitted assessment materials. For all facets, agreement was at an acceptable level of 70% or above.

Teachers' Performances

Since the participating teachers worked in teams throughout the semester, the assumption of independence required for most statistical tests could not be met. One common approach for dealing with this problem is to use the team rather than the individual as the unit of measure. However, several of the teachers changed teams during the semester, resulting in ill defined team units. Therefore, the unit of measure used here is the individual and the analysis is restricted to descriptive statistics.

The average score for "Standard" across teachers for each of the four submissions was 3.71, 4.00, 3.79 and 3.96. The initial high rating with respect to this category is not surprising given that the assessment portion of the workshops had already addressed the importance of aligning tasks with the algebra standard prior to the first submission. Additionally, many of the participating teachers had a great deal of teaching experience and this provided them with prior experiences in the selection and development of assessment tasks. Since the majority (23 out of 28) of the teachers had reached the categories' ceiling on their first submission, the initial high performance of the teachers with respect to this category at the start of the project may have prevented the identification of further growth. By the final submission, 27 out of the 28 teachers submitted a task that was rated as a "4".

With respect to the "Criteria" category, the average scores for each submission was 3.39, 3.79, 3.75 and 3.79. This category examined the extent to which the requirements of the submitted rubric were aligned with the requirements that were set forth in the task. In this category, there was an increase in the average score from the first submission to the second submission. As was discussed earlier, the focus of the second workshop was upon creating scoring rubrics that were consistent with the nature of the task. The increase that followed the second workshop suggests the effectiveness of these efforts. After the second submission, the average score with respect to this category remained relatively stable on the remaining submissions, suggesting the sustainability of the teachers' efforts.

The average score for the "Language Specificity" category for each of the four submissions was 3.50, 3.68, 3.75 and 3.79. This suggests a gradual improvement with respect to "Language Specificity" across time. The primary emphasis on this category was in the second and third workshop, which took place prior to the third submission. The continued increase within this category on the fourth submission suggests that the teachers were able to continue to improve with respect to this category without additional instruction.

The final category was "Feedback to Students". The average scores with respect to this category across the four submissions were 2.11, 2.46, 2.61 and 2.57. As these numbers suggest, there was an increase from the first to the third submission. After the third submission, there was a decrease. The primary emphasis on this category occurred in the second and third workshops. This is consistent with the observation that the teachers' average score peaks on the third submission. There was a slight decrease with respect to this category on the fourth submission. This raises to question whether changes in performance within this category are sustainable without further workshop intervention. A more extended study would be needed to test the accuracy of this claim. The final value did not drop below the average score on the first submission.

The average score for the "Total Score" across submissions was 12.71, 13.93, 13.89 and 13.93. Based on the average scores, the first and second workshop appeared to have the strongest influence on the teachers' assessment knowledge. The reader may question the practical significance of a "Total Score" that increased from an initial value of 12.71 to a final value of 13.93. However, if these scores are scaled to be out of a hundred, this results in a difference between receiving a "Total Score" of a 79 and an 87.

Discussion

The purpose of this article is to present the methods and results of the assessment portion of the C^2 project. This project represents a collaborative effort between a local school district and university to improve the instruction that students receive in mathematics. As part of this project, the CAMS rubric was developed and used to evaluate the teachers' submitted assessment materials. This rubric was specifically designed to assess the following components of the teachers' materials:

1. The consistency of the assessment task to the algebra standard ("Standard");

2. The consistency of the rubrics scoring criteria to the task requirements ("Criteria");

3. The extent to which the teacher proposed scoring rubric clearly distinguishes score categories ("Language Specificity"); and,

4. The amount of feedback that the teachers provided to students ("Feedback to Students").

As was illustrated in this paper, the use of the CAMS rubric did provide reliable results. This was accomplished through the careful training of the raters. Additionally, the observation that the average score for the teachers in the categories of "Criteria", "Language Specificity" and "Feedback to Students" increased immediately after the pedagogical workshops that addressed the related objective suggests that the rubric was successful in capturing the anticipated changes.

Very little change was witnessed across submissions with respect to the "Standard" category. The majority of participating teachers had reached the ceiling with respect to this category on their first submission. Since in-service teachers already have classroom experience in the creation of tasks that are consistent with their goals, they may not need additional assistance with respect to this category. Additionally, the participating teachers were working in teams and through these teams, the inexperienced teachers had immediate access to their more experienced peers. The experienced teachers may have ensured that the developed tasks were aligned with the stated goals. Another influencing factor is that the discussion that concerned aligning tasks to the algebra standard occurred during the first workshop, which was prior to the first submission. This too probably influenced the high scores that were witnessed with respect to "Standard" on the first submission. If the CAMS rubric had been used to evaluate the assessment materials of pre-service teachers, the initial score with respect to "Standard" would probably have been much lower.

A disappointing component of this study is that the participating teachers continued to provide their students with limited feedback at the conclusion of the study. Although their average performance had increased, it still fell between level "2" and "3". Since feedback provides students with the opportunity to improve, providing appropriate feedback is essential to the learning process. Another concern is that the teachers' average performances with respect to feedback decreased from the third to the fourth submission. This may suggest that in order for teachers to improve with respect to this category, they need on-going instruction concerning the importance of providing students with detailed feedback. It is also possible that the participating teachers did have had the knowledge that was necessary to provide appropriate feedback to their students, but lacked the time that was necessary to use this knowledge. Future studies are needed that explore methods for encouraging teachers to provide detailed feedback to their students.

The proposed CAMS rubric is likely to be useful well beyond the confines of this study. Researchers have argued that both in-service and pre-service teachers require additional training with respect to classroom assessment (Brookhart, 1999; Stiggins, 1999; Whittington, 1999). This training is likely to include an emphasis on the importance of assessment tasks being aligned with a set of standards and the use of scoring rubrics for classroom assessment purposes. The CAMS rubric provides an instrument for evaluating teachers' assessment materials with respect to these areas. Although this rubric was designed to be aligned with the Jefferson County Algebra Standard, it can be easily adapted to any given standard within a given content area. The minor changes that would need to be made are as follows: 1) altering the references to the algebra standard to a reference of the standard of interest, and 2) altering the references to mathematics to referencing the subject of interest.

This study also provides an example of the types of programs that can be developed for in-service teachers when school districts work with universities. The Jefferson County School District had an expressed desire to improve their mathematics teachers' assessment capabilities. The Colorado School of Mines had the appropriate experts in both mathematics and assessment to assist Jefferson County in their efforts. For more information concerning the larger goals and structure of C^2, see Moskal and Bath (2001a; 2001b).

References

Brookhart, S (1999). "Teaching about Communicating Assessment Results and Grading." Educational Measurement, 18 (1), 5-13.

Jefferson County School District (1996). "Mathematics Content Standards" [On-line]. Available: http://204.98.1.2/isu/math/mathstan.pdf.

Mertler, C. (2001). "Designing Scoring Rubrics for Your Classroom. Practical Assessment, Research & Evaluation, 7 (8) [On-line]. Available: http://ericae.net/pare/getvn.asp?v=7&n=25.

Moskal, B. & Bath, B. (2001a). "A Response to the NCTM Standards: Confidence and Competence Project (C^2)." Academic Exchange Quarterly, 5 (4), 203-211.

Moskal, B. M. & Bath, B. B. (2001b). "Combining Expertise to Create an Effective Teacher Development Program: The Confidence and Competence Project (C^2)". Colorado Mathematics Teacher, 34 (1), 7-12.

Stiggins, R.J. (1999). "Evaluating classroom assessment training in teacher education programs." Educational Measurement, 18 (1), 23-27.

Whittington, D. (1999). "Making room for values and fairness: Teaching reliability and validity in the classroom context." Educational Measurement, 18 (1), 14-22.

Barbara M. Moskal, Colorado School of Mines

Jill Fellman, Jefferson County School District, CO

Barbara B. Bath, Colorado School of Mines

Dr. Moskal is an Assistant Professor in Mathematical and Computer Sciences and the Associate Director of the Center for Engineering Education. Ms. Fellman is the Director of Instructional Services in the Jefferson County School District. Dr. Bath is the Director of Undergraduate Studies and an Associate Professor in the Mathematical and Computer Sciences Department.