Understanding how teachers engage in formative assessment.
As a result of the ongoing standards movement and No Child Left Behind, there is a plethora of data available to teachers today. Teachers have access to standardized test scores, interim assessment outcomes, and data they gather as part of their daily teaching practice. There is so much data, however, that teachers can become overwhelmed. Frequently they do not know how to interpret or change their teaching practice in response to those data. Some field experts think formative assessment could be a solution (e.g., Stiggins, 2004). Still, for formative assessment to help teachers change their classroom practices, they must learn about formative assessment--and that is where professional development (PD) comes in.
Our knowledge of the subject-specific nature of high quality PD suggests there are important learning challenges that teachers face as they engage in content-specific professional development (e.g., Cohen & Hill, 1998; Wilson & Berne, 1999). Those challenges are related to the particular cognitive demands reform-minded teaching requires (e.g., Smith, 1996). We begin with the assumption that learning formative assessment also has particular cognitive challenges. For example, teachers must learn how to systematically collect evidence of all students' learning as that learning is happening in real time. Once teachers begin collecting such evidence, they must learn new ways of pacing, differentiating, organizing, and adapting their instruction in order to adequately address the range of student thinking in the classroom. This paper informs increasing calls for formative assessment (i.e., Black & Wiliam, 1998b; Popham, 2008; Stiggins, 2004) by describing the changes teachers saw in their professional practice as they were in the process of learning formative assessment. We discuss the implications of the changes for PD leaders. Specifically, we discuss the kinds of support teachers need as they begin to learn about formative assessment.
What is Formative Assessment?
Formative assessment continues to receive increased attention in the field of education as being a cost-effective method of improving student learning (Black & Wiliam, 2007). However, defining formative assessment is problematic since it is often viewed as any use of assessment to support instruction. In fact, when teachers hear about formative assessment for the first time, they often say, "I do that already." We define formative assessment as a process a teacher uses to elicit evidence of student learning that is analyzed and used to adjust instruction to better meet student learning needs. This vision of formative assessment involves more than adding "extra" assessment events to existing teaching and learning. It also requires teachers to use the information they collect to modify instruction. In classrooms where formative assessment is used with the primary function of supporting learning, the divide between instruction and assessment becomes blurred. Thus, formative assessment is an ongoing, cyclical process woven into the life of the classroom (Thompson & Wiliam, 2007).
For example: A fifth-grade class might be learning about equivalent fractions. At the end of the lesson, a teacher asks her students to write on a note card two fractions that are equivalent to 2/3. She then collects their note cards as an "exit ticket" when they leave the room. Information gained from this simple activity is then analyzed by the teacher and used to adjust her next day's instruction accordingly. If most of the students were not able to correctly identify two equivalent fractions to 2/3 (i.e., 4/6, 6/9, 8/12, etc.), the teacher could review equivalent fractions with the entire class. However, if most of the students were able to identify two equivalent fractions, the teacher could move on to converting fractions into decimals or move on with only those who understood and review with those who did not. While the quality of large-scale assessments is determined on the basis of reliability and validity, the quality of formative assessment takes on a somewhat different appearance. In this example, the quality of the exit ticket question can only be determined in the light of the teacher's goals for both the lesson and where this lesson fits in the larger sequence of instruction. By eliciting ongoing evidence of every student's learning in quick and simple ways, teachers have the ability to immediately assess students' needs and make timely adjustments necessary to instruction. Given the very low-stakes of this assessment, any misclassification can be quickly noticed and remedied by the teacher.
The point of this example is to demonstrate for the reader that when formative assessment is properly implemented it is not strictly about instructional strategies (e.g., the use of exit tickets in this example). Formative assessment requires teachers to deliberately elicit evidence of student thinking, make decisions about what to do with that evidence, and then implement appropriate changes in instruction. This is not done every six or nine weeks. It is done every day for the entire school year.
The Keeping Learning on Track Program[R]
Based on this view of formative assessment, a professional development program for teachers called the Keeping Learning on Track Program[R] (KLT) was developed. KLT is a two-year, sustained, interactive PD program that supports teachers to adopt minute-to-minute and day-by-day formative assessment strategies shown by research to increase student learning (see Wiliam, 2007, for the research base behind KLT). We define student learning, gleaned from our ongoing research, as higher scores on standardized tests; content knowledge growth assessed by classroom evaluations, projects, and daily activities; heightened student engagement and motivation; and increased student ownership and accountability for their work (Black & Wiliam, 1998a; Tocci & Taylor, 2007; Wylie, Lyon, Ellsworth, & Martinez, 2007; Wylie, Thompson, & Wiliam, 2007). Teachers are introduced to the KLT program and learn about the program's organizing principle. This principle is "Students and teachers using evidence of learning to adapt teaching and learning to meet immediate learning needs minute-to-minute and day-by-day" (Thompson & Wiliam, 2007, p. 6). Five strategies are discussed and explored to help teachers think about how to implement KLT in their own classrooms and to regulate the learning process:
Sharing Learning Expectations. Teachers explain and clarify learning intentions and criteria for success with the class in student friendly language--not content standard language. For example, success criteria may include sharing exemplars of other students' previous year's assignments with the current class and/or having students critically assess the quality of a sample assignment before they complete a similar assignment.
Questioning. Teachers engineer effective classroom discussions, questions, and learning tasks that elicit evidence of learning. This does not mean teachers are simply listening for "correct" answers. Rather, teachers design meaningful questions that engage all students (e.g., asking a question and then randomly selecting a student name out of a jar to answer) with the specific intent of encouraging students to think on deeper levels (e.g., questioning to promote critical thinking skills) or to provide teachers with information about student understanding (i.e., questioning to uncover student misconceptions).
Feedback. Teachers provide feedback that moves learners forward. Simply providing students with a grade and positive comments such as "Nice Job!" or negative comments such as "You can do better than this" does not help students improve upon their work. Instead, teachers are taught to provide actionable feedback which focuses on providing a "recipe" for improvement. For example: 'Tou have 3 punctuation errors in the first paragraph of your paper. Find them, fix them, and then show me." This type of feedback promotes higher order thinking and provides support, structure, and time for students to make corrections.
Activating Self. Teachers activate students as the owners of their own learning. Teachers give up some of their control in the classroom (e.g., pacing of the lesson or the topics/content covered or reviewed, etc.) to allow students the ability to assess their own understanding of the material. For example, each student is provided with red, yellow, and green cups. During the lesson, students display their level of understanding (green represents, "I get it!," yellow represents "I'm not sure," and red signifies "I don't understand"). The teacher gauges the class's understanding and moves on, slows down, or reteaches content as necessary.
Activating Peers. Teachers activate students as instructional resources for one another. Students give and receive feedback to and from one another. Using the example above, if a student indicates he/ she does not understand, the teacher may call upon a student displaying a green cup to explain the answer or concept to students showing red or yellow cups. Not only does this continue to engage the entire class (students need to be prepared as they do not know if and when they will be called on to help), it provides additional information about whether those indicating "I get it" really understand.
In addition to the research-based framework provided by the five strategies, teachers are also exposed to a wide range of assessment for learning (AfL) techniques that contextualize the strategies. The techniques represent specific, concrete ways a teacher might choose to implement one or more of the strategies. Table 1 provides some descriptions of AfL techniques. Leahy, Lyon, Thompson, & Wiliam (2005) describe a more comprehensive overview of AfL techniques. To meet the varying needs of teachers, techniques are low tech, low cost, and appropriate for use across grade levels and subject matters.
The ideas described above are presented to teachers through introductory workshops and supported long term via teacher learning communities (TLCs) over two years. TLCs allow the content of KLT to become locally adapted (Cobb, McClain, Lamberg, & Dean, 2003), over a period of time (Cohen & Hill, 1998), with teachers working as collaborative partners (Garet, Porter, Desimone, Birman, & Yoon, 2001; Ingvarson, Meiers, & Beavis, 2005). Literature on TLCs further suggests the content of those meetings is crucial to what teachers learn from the communities (Borko, 2004; Borko, Mayfield, Marion, Flexer, & Cumbo, 1997; Elmore, 2002; Kazemi & Franke, 2003; McLaughlin & Talbert, 1993; Putnam & Borko, 2000). Therefore, teachers use their time in TLCs to share and discuss with other teachers how they will or have implemented specific AfL techniques, write an action plan to try something new or try again in a different way, and report back next month. A main focus of TLCs is to help teachers learn what to do with all the new information they have about what their students do and do not understand. To help teachers understand how TLCs work and to prepare a subset of teachers to lead a TLC, the program also provides a two-day TLC Leader workshop. TLC Leader workshops outline the research behind TLCs and the importance of ongoing, guided learning, practice, reflection, adjustment, and supportive accountability as teachers take on implementing KLT in their classrooms.
We agree with the notion, advanced by many in the professional development community, that cognitive demands of particular reforms shape teachers' reactions to those reforms (e.g., Cohen & Hill, 2001; Spillane & Thompson, 1997; Spillane & Zeuli, 1999). In order to better understand the challenges associated with teachers learning formative assessment, two questions frame our analyses. First, how do teachers perceive formative assessment impacts their professional practice? Second, what does this mean for formative assessment professional development?
During the 2006-2007 school year, teachers from a large metropolitan Midwest school district participated in the KLT program. In this analysis we focused on 65 teachers who voluntarily completed an open-ended evaluative questionnaire. The group of teachers was 65% female and 18% male, with 17% not reporting gender. All teachers taught in K-8 buildings during the 2006-2007 school year. Teachers attended the initial two-day workshop introducing them to the research and concepts behind KLT and TLCs. To varying degrees, they also participated in up to seven school-based, monthly TLC meetings over the course of a year. Participants reported attending an average of five TLC meetings, with a range of one to seven.
Because we do not have data on all the teachers who participated, we do not know how the 65 teachers who returned the questionnaire compare with the other teachers who participated in the program. However, based on the responses we did receive, we know that among the 65 there was a wide range of participation and support for the program. Given the constraints of our sample, we describe the range of ways teachers understood formative assessment but do not generalize to suggest certain understandings will be more or less prevalent among teachers.
Data Collection and Analysis
Open-ended questionnaires evaluating the KLT program were mailed to TLC teacher leaders at each participating school. Leaders distributed and collected the surveys during the final TLC meeting. The questionnaire consisted of four demographic items and eight open-ended questions pertaining to teachers' impressions of their learning, changes they saw in the classroom, personal thoughts, student thoughts, expectations, and challenges of the program (see the Appendix for survey instrument).
Responses were entered into N<5, a qualitative software program, and coded. Based on the literature described above, the content of the questionnaire, and researchers' experience working with TLCs, initial axial codes were developed and used to code the responses. These early codes focused on either teacher or student outcomes. Teacher outcome codes were the outcomes and goals teachers held or realized were important for themselves while in the KLT program. Teacher outcome codes consisted of: collect evidence, share expectations, student ownership, student collaboration, feedback, adapt instruction, and teacher collaboration. Although student ownership may seem counterintuitive as an initial code for teacher outcomes, it was initially placed here because it represented a teacher's desire to "give up" some of their control of the classroom and allow students to take more ownership over their own learning. Therefore, originally we considered this to be a teacher outcome as the focus was on the teacher's goal of "letting go." Student outcome codes were the outcomes teachers saw resulting from their normative assessment strategies. Student outcome codes were: engagement, peer support, learning responsibility, and improve learning. After a preliminary round of coding, memos were written and discussed among the researchers, and codes were modified to reflect what was found in the responses (Miles & Huberman, 1994). Some codes were merged, others refined, and still others created. The final coding scheme consisted of three main themes teachers felt were relevant to their learning formative assessment: support system, student practice, and teacher practice. For more detailed information on the coding process see Figure 1.
[FIGURE 1 OMITTED]
Teachers indicated that learning about formative assessment impacted three aspects of their professional practice: (a) the nature and existence of a school-based, professional support system, (b) the ways in which their students engaged in learning, and (c) the nature of their teaching practice. There were a range of responses in each of the aspects teachers identified. For example, teachers reported gaining greater support of their teaching practice as well as personal support due to their involvement and participation in TLCs. Teachers also noticed student practices were impacted as students participated more, became more accountable for their learning, and held perceptions of fairer classroom practices. Finally, teachers found their own practice to have been affected by formative assessment through the use of AfL techniques and decreased classroom management issues. Table 2 summarizes these findings. The changes teachers experienced in their classroom support a "learning by doing" model, where repeated cycles of learning, practice, reflection, and adjustment facilitated teachers' learning of formative assessment (Thompson & Wiliam, 2007).
Two thirds of the participating teachers (n=43) felt collaboration and the sharing of ideas and techniques about how to use formative assessment in the classroom while at TLC meetings positively impacted their development. The support system they built through the TLCs allowed for reflection time and access to ideas they would not have thought of themselves. The following examples illustrate how teachers discussed this:
* The sharing has helped a great deal! The modeling of the lessons has benefited a great deal. There have been some great suggestions or strategies; techniques that I've never heard of or never considered.
* Sharing evidence of learning (at TLCs) was helpful to [sic] re-evaluate my own success and failures. It helped me to make priorities and reinforce what is (was) working.
Continued learning about formative assessment in TLCs supported teachers' development of instructional techniques, but it also supported them personally. TLCs provided some teachers with an overarching feeling of "team" or "togetherness." No longer did teachers have to work alone. They now had a support system they could access for assistance, encouragement, and motivation. Teachers explained,
* It [TLC meetings] meant we can say our thoughts and not be judged. I felt that my co-workers listened to me ... I felt enriched. I realized I didn't have to know everything. I had co-workers that would help me through anything and I would do the same for them. I felt encouraged to try new things because others had tried them and they worked.
* The TLC meetings I felt were beneficial and we held great discussions ... the team was strong and extremely supportive.
These types of comments suggest that teachers viewed the emotional support TLCs provided as important to their desire to take the risks necessary to learn about and implement formative assessment.
Fifty-three of the 65 participating teachers (82%) suggested various positive impacts on student practice as a result of their efforts implementing formative assessment in their classrooms. Teachers frequently indicated increased student participation, students becoming more accountable for their own learning, and student perceptions of fairness in the classroom as outcomes of using formative assessment in the classroom.
Student participation. Forty-five teachers (69%) indicated that student participation increased during the time they were implementing formative assessment in their classrooms. Teachers noticed two areas of change in student participation: perceived student attitudes and observed student actions. The relationship between student attitudes and actions was clear in teachers' responses. These representative examples illustrate how teachers believed student attitudes toward AfL techniques increased student participation in class:
* I had students comment on how much they enjoyed Two Stars and a Wish. They stated the stars made them feel good and the wish was something they really tried working on. Before Two Stars and a Wish, the kids said they never read the comments, just looked for the grade.
* I almost use KLT daily with Trophies in Language Arts. Popsicle[R] sticks, dry erase (boards), ABC (cards) ... all make students excited about their work and they want to participate! Very helpful every day!
There were, however, a handful of teachers (n=4) who described positive student attitudes but did not link those attitudes to student actions. Examples include "The Popsicle[R] sticks and white boards are so easy to use and the kids like to use it--a novelty!" and "The students liked using the white boards."
Teachers also noticed how formative assessment shaped student participation with no preface to student feelings (n=21). For example, one teacher explained:
* We were reviewing the life cycle of a butterfly. I was using Red cards/Green cards for understanding. At one point I thought the students knew the difference between a caterpillar and a chrysalis until the students held up the Red cards because they didn't know the difference.
In this situation, had the students not been actively involved in the lesson, they would not have held up their Red cards to show the teacher they needed further explanation for better understanding of the life cycle. However, the teacher did not reference how the students felt about using Red cards.
In addition to specific examples of student participation, a quarter of the teachers (n=16) reported an interaction between participation and students taking responsibility for their own learning. In these situations, teachers commented that students were actively participating in classroom activities, thus enabling students to become more responsible for their own learning. For example:
* They loved having their own white board ... the program created positive peer pressure and the students helped keep their own learning on track. They also felt more actively involved in their own assessment.
Teachers shared a variety of ways they experienced their students' participation increasing in the classroom. Whether it was through new-found student excitement for learning, higher levels of student involvement, or both, teachers believed their students' participation improved when they were using AfL techniques.
Student accountability. In addition to changes in students' attitudes and actions, twenty-six teachers (40%) noticed that students took more responsibility or ownership for their learning as a result of using AfL techniques. Teachers saw this as a direct outcome of providing students with opportunities to do so. Sharing expectations (n=6), increasing student responsibilities (n=9), and/or implementing AfL techniques in the classroom (n=14) are ways teachers noticed learning becoming more of a student responsibility. Examples of each are listed in respective order:
* Two stars and a wish--seeing/observing the improvement in their (the students') writing skills by highlighting the "wish"--students had a clearer goal of what to accomplish in a particular writing assignment ... they could not wait to "fix" their writing in order to fulfill the "wish."
* I assign jobs to my children and responsibilities for their own learning ... they have learned to check each other's written work by editing and revising ... working on giving them more responsibility for reciprocal teaching in reading.
* They [students] ask me if they can get the whiteboards and practice their work. They encourage one another and work in groups. They like taking responsibility for their own learning.
The quotations above demonstrate the relationships teachers saw between students' acts of participation and student accountability for learning. As students participated in class more, they often also began taking on more ownership for their learning.
Students' perceived fairness. Thirteen (20%) teachers brought up students' perception of fairness in the classroom as an impact of implementing formative assessment. This theme caught our attention and warranted further investigation because all of these responses referred to the No Hands Up AfL technique as the driving force behind students' perception of fairness. No Hands Up is a set of AfL techniques in which teachers ask a question for all students to think about, wait an appropriate amount of time, and then choose a student to respond via a method of random selection. Such techniques gave students the impression that "it's fair because everyone gets a chance" and "they didn't think I [the teacher] was playing favorites" with "the same kids being called on." Not only did every student have the opportunity to be selected to answer questions, almost a third (n=21) of the teachers reported "they [students] knew that they needed to better focus on the task because they could be called upon to answer" and "it kept them [students] on their toes" or paying better "attention" to the lesson.
Teachers saw a number of different changes in their students when using formative assessment in the classroom. Some changes were attitudinal, others were behavioral. As we discuss later, changes in students' attitudes and behaviors may have important implications for teachers' engagement with formative assessment.
Teachers described two ways their classroom practices were influenced by learning about formative assessment. Their comments focused on formative assessment use and changes in their classroom management.
Formative assessment use. Almost all of the teachers (n=60; 92%) reported that actually trying to implement AfL techniques introduced in the KLT professional development program had a positive impact on their practice. There was, however, differential use of implementation among teachers.
In order to better understand the ways in which teachers thought about and used formative assessment, we coded teachers' responses into levels. At the lowest level, Collect Evidence, teachers used formative assessment to provide an opportunity for data to be collected about student understanding. Teachers implemented AfL techniques but did not apply them effectively. At the next level, Collect Evidence and Analyze, teachers analyzed the evidence they collected. We view this as a higher level of formative assessment use because the teacher now gains a clearer picture of what students understand. However, they are still not using the techniques to their fullest potential. Finally, at the highest level, Collect Evidence, Analyze, and Adapt Instruction, teachers utilized the full effectiveness of AfL techniques by using the results from collected and analyzed data to adapt or influence future instruction based on student needs. Although the coding scheme for formative assessment use is intended to differentiate between those teachers who reported using AfL techniques to the fullest extent and those who did not, we must acknowledge that we are unable to determine whether the level reported was the highest or lowest level ever used in the classroom. As is the case with all self-report data, there is always the possibility that self-report over- or underestimates specific features of teaching practice. That said, visits to classrooms using these self-report measures in other studies suggest teachers' self reports are not far off what external observers witnessed during classroom visits.
In the following paragraphs we document how teachers used AfL techniques at different levels. Most often, a single teacher's response fit into a single category. There were, however, a modest number of teachers who reported more mixed practices.
Instances where teachers reported using AfL techniques at the lowest level, Collect Evidence, were rarely (n=7) found in the responses. Yet some teachers did suggest the use of AfL techniques at this low level. For example,
* I took pictures of the students working on the dry erase boards. I also took pictures of kids using Popsicle[R] sticks.
* I use the exit question as we walk out the door to the restroom or lunch.
In both instances, teachers suggested they were using AfL techniques and collecting evidence, yet they do not elaborate as to whether they actually analyzed the data received or did anything with the information to adapt instruction.
Many teachers (n=29) reported using formative assessment in their classroom at the Collect Evidence and Analyze level. Examples are the following:
* The answer cards that I used (ABCD) allowed me to see how students are making choices during multiple choice tests in reading and math.
* I use thumbs up and thumbs down. I can see instantly which kids grasp the material and which ones don't.
Both cases suggest the teachers collected evidence of learning and took the time to analyze the data; however, they do not mention if they adjusted their instruction based on this information.
More than half of the teachers (n=32) who reported that using AfL techniques in their classroom impacted their practice reported implementation of techniques at the highest level, Collect Evidence, Analyze, and Adapt Instruction. At this level, teachers went beyond thinking about evidence of student learning to actually changing instruction. For example,
* In a math lesson on dividing with decimals, after the lesson, a quick problem is given as an exit prior to lunch. I look over the boards while students are at lunch for success. When they return, we either review if low success rate or move on to assignment if high, and I am able to work with small struggling group.
* The use of white boards permitted me to see which students were struggling and which students understood. I would then move on with those that understood and reteach for the others.
Though we do not know exactly how teachers retaught or how skillful and appropriate their instructional adjustments were, these types of comments suggest some teachers were changing instruction in response to collected evidence of student learning. This pattern emphasizes again the importance of students in teachers' learning and implementing formative assessment.
Classroom management. In addition to actually trying AfL techniques, twenty-two teachers (34%) also thought that their classroom management was positively impacted as a result of using AfL techniques in their classrooms. Some teachers (n=7) felt that the use of formative assessment had a general impact on their classroom management by adding structure and helping with discipline while others (n=6) attributed the improved classroom management to students knowing what was expected of them. Teachers (n=14) cited specific AfL techniques (i.e., Popsicle[R] sticks and whiteboards) as particularly helpful to "minimize loud talking" or "decrease the transition time."
Although attributing improved classroom management to the use of formative assessment may be a positive outcome for many of the teachers, it is likely that in some cases the techniques were misused and did not maintain fidelity to the KLT professional development program. For example, one teacher said, "I can move a lot faster when I don't have to select students to help--Popsicle[R] sticks." This statement suggests this teacher is not using the Popsicle[R] sticks technique formatively. She emphasizes how quickly she can work when using Popsicle[R] sticks to select students. Other teachers reported similar misuses of AfL techniques which led to improved classroom management:
* I used Popsicle[R] sticks to keep track of who I called on in class.
* The techniques helped to make the day go smoothly. The Popsicle[R] sticks helped decrease the transition time.
Popsicle[R] sticks and the No Hands Up techniques, more generally, are meant to give all students time to think and prepare an answer knowing they might be called on. The goal of the technique is to provoke thinking and ownership of learning among students. Descriptions like these suggest that teachers may not be using AfL techniques as intended, even when they report they are doing formative assessment. We return to this point and its implications in the following sections.
It seems somewhat obvious that unless teachers feel they are learning something worthwhile and relevant to their practice in professional development (PD) they will not continue voluntarily. Or, perhaps worse, they will continue and simply wait for the program to end, never really engaging the ideas and discouraging use, causing the group to suffer for their disinterest. Our findings suggest there may be an interesting mechanism that reinforces teachers' desire to continue learning about formative assessment. As we described above, teachers noticed their students perceived the new AfL techniques to be more "fair." Teachers also noticed how the techniques helped students take more accountability for their own learning and enjoying the work they do. This created a learning environment in which more students were engaged. If it is true that as engagement increases learning increases (e.g., Cobb, 1972; Hecht, 1978), we hypothesize that such an environment is encouraging to teachers. Positive feedback from students (with whom teachers spend the vast majority of their day) may support teachers' desire and willingness to learn more about formative assessment and implement new strategies and techniques in their classrooms.
The same may be true of impacts teachers report about classroom management. While improved classroom management may be a sign formative assessment is not being used as it was intended (teachers eliciting evidence of student thinking more regularly and adjusting instruction), it may also mean teachers have not understood formative assessment accurately. If, for example, teachers only ever understand formative assessment as techniques that can make students behave better (i.e., "keep them on their toes"), the deeper goals of formative assessment will go unmet. Evidence of student learning will not be analyzed and used to change instruction, and the traditional role of teacher as "sage on the stage" will continue.
Regarding support systems, all of the teachers reported that the TLC meetings had a positive impact and were beneficial in furthering their understanding of formative assessment. This relates back to the literature cited earlier in this paper. Teachers need supports and structures to try new things in their classroom and reflect upon those experiences within the context of their instruction in order to make changes to their practice. Learning alongside their peers, colleagues can grow together to build a supportive community that allows teachers to ask questions, seek help, or support and provide feedback to others.
The qualitative data upon which this study draws are helpful for pointing out teachers' concerns and the range of issues PD leaders will face. However, the study is limited in specific ways. First, this study draws exclusively from teacher self-reports. Although a number of studies have assessed the validity of instructional measures based on teacher self-reports (Burstein et al., 1995; Mullens & Gayler, 1999; Mullens & Kasprzyk, 1996, 1999; Porter, Kirst, Osthoff, Smithson, & Schneider, 1993), we recognize relying solely on this type of data limits our claims and external validity. Also, the questionnaires were anonymous so we could not do any member checking to assure ourselves we were interpreting teacher responses as they intended. Additionally, the anonymous nature of the data did not allow us to connect responses to classroom and TLC observations we conducted. Again, drawing solely from one source of data--in this case, questionnaire responses--limits our ability to make claims about what actually occurred in each of the teacher's classrooms (Camburn & Barnes, 2004; Mayer, 1999). As such, we have steered away from these types of claims. Finally, since we only heard from a subset of teachers who volunteered, findings may not be representative of the entire sample of participants introduced to the program. Despite these limitations, these data help us understand how teachers respond to learning about formative assessment and suggest the specific issues PD leaders might pay attention to as they work to support teacher learning.
Just as teachers need to understand students' ideas about fractions or double-digit subtraction in order to help them learn mathematics, professional development leaders need to know what teachers understand about assessment if they are to help them learn. Professional development leaders need to know the particular pitfalls common to learning how to use assessment formatively. They also need to understand how classroom dynamics change when teachers begin to use evidence of student learning in more systematic ways.
These data suggest a number of implications for district leaders and professional developers as they plan supports for teachers who are learning formative assessment. One major implication for PD developers concerns student reactions to formative assessment. When students give teachers positive feedback through improved attitudinal and/or behavioral changes, powerful learning incentives for teachers are created. Teachers' motivation to continue using and/or refining their formative assessment practice is reinforced. Since teachers reported student attitudes and behavior as important factors in their learning of formative assessment, professional development leaders need to pay careful attention to the relationship between student reactions and teachers' desire to continue learning as they try and support teachers learning formative assessment. In this study, teachers reported positive student reactions to AfL techniques. But there may well be negative reactions. PD leaders should be keenly aware of the student reactions teachers describe as they may shape teachers' willingness to try other AfL techniques.
Professional learning also requires scaffolded practice over time. TLCs provided a safe learning environment where teachers learned about and discussed their implementation of AfL techniques and learned to plan effective classroom discussions, questions, and tasks which would elicit student thinking. Having a peer support group to play out instructional sequences prior to their occurrence allowed teachers to anticipate the wide range of responses students might have. They were also able to plan ahead of time what to do next. As a result of teachers' strong belief that these TLCs provided professional and personal support for their learning and implementing AfL techniques in the classroom, it is important for PD leaders to actively protect teachers' learning time and be very mindful of individual teachers' learning during that time.
Using formative assessment in the classroom fundamentally changes how teachers think about their roles. Eliciting evidence of student learning and modifying instruction as a result of that evidence deeply threatens the "sage on the stage" metaphor that some teachers see as their role. In the "sage" view, confidence comes from knowing the subject matter, telling it to students, and occasionally checking to make sure students acquired that knowledge. In a formative assessment view, confidence comes from knowing how to routinely elicit, analyze, and utilize the range of student understandings that are now a part of the classroom conversation. In both views, teachers must understand the subjects they teach, but they must understand them in very different ways. If PD leaders are to effectively help teachers learn about formative assessment, they must identify and develop ways of supporting the subjectspecific learning teachers must do as student thinking becomes more visible and students increasingly take responsibility for their own learning.
As our paper demonstrates, teachers' feedback tells us what keeps them coming back to learn more, but it also suggests the areas of practice that may require particular attention from PD leaders. If formative assessment is to have the impact advocates suggest, leaders will need to pay careful attention to how student reactions, time, and teacher peer support groups are influencing teachers' learning. Only then will teachers have the opportunity to be listened to, supported, and challenged in ways that will help them improve.
Keeping Learning on Track Teacher Survey
Your school district worked with ETS to roll out Keeping Learning on Track (KLT) during the 2006-07 school year in an additional five schools. Part of our job at ETS involves us in evaluating our own work and progress and seeking to make improvements. The following questions are to help us learn about what aspects of last year's work were most memorable and important to you. Your information will not be shared with anyone in a way that would identify you as an individual.
1. Describe your biggest learning "aha" as a result of Keeping Learning on Track.
2. What is the greatest area of lingering confusion or uncertainty?
3. Describe an impact that the Keeping Learning on Track project had on your DAILY practice. Provide a specific example.
4. Describe an impact that the Keeping Learning on Track project had on how you collect and use evidence of student learning. Provide a specific example.
5. Did students comment on any changes in your classroom over the course of last year (changes that you attribute to your involvement with Keeping Learning on Track?) Give a specific example of the change and the student comment.
6. How beneficial, and in what ways, were the teacher learning community meetings (school-based meetings)?
7. Describe any aspects of the project that did not meet your expectations. How might your school, district, or ETS staff address the problem?
8. Did you have adequate support for facilitating the monthly TLC meetings? If not, what would you have needed?
Black, P., & Wiliam, D. (1998a). Assessment and classroom learning. Assessment in Education: Principles, Policy and Practice, 5(1), 7-73.
Black, P., & Wiliam, D. (1998b). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80(2), 139-148.
Black, P., & Wiliam, D. (2007). Large-scale assessment systems: Design principles drawn from international comparisons. Measurement: Interdisciplinary Research and Perspectives, 5(1), 1-53.
Borko, H. (2004). Professional development and teacher learning: Mapping the terrain. Educational Researcher, 33(8), 3-15.
Borko, H., Mayfield, V., Marion, S., Flexer, R., & Cumbo, K. (1997). Teachers'developing ideas and practices about mathematics performance assessment: Successes, stumbling blocks, and implications for professional development (CSE Technical Report 423). Los Angeles, CA: CRESST.
Burstein, L., McDonnell, L. M., VanWinkle, J., Ormseth, T., Mirocha, J., & Guiton, G. (1995). Validating national curriculum indicators. Santa Monica, CA: RAND.
Camburn, E., & Barnes, C. A. (2004, September). Assessing the validity of a language arts instruction log through triangulation. The Elementary School Journal, 105(1), 49-73.
Cobb, J. A. (1972). Relationship of discrete classroom behaviors to fourth-grade academic achievement. Journal of Education Psychology, 68(1), 74-80.
Cobb, P., McClain, K., Lamberg, T., & Dean, C. (2003). Situating teachers' instructional practices in the institutional setting of the school and district. Educational Researcher, 32(6), 3-24.
Cohen, D., & Hill, H. (2001). Learning policy: When state education reform works. New Haven, CT: Yale University Press.
Cohen, D., & Hill, H. (1998). State policy and classroom performance. Philadelphia, PA: University of Pennsylvania Consortium for Policy Research in Education.
Elmore, R. (2002). Bridging the gap between standards and achievement: Report on the imperative for professional development in education. Washington, DC: Albert Shanker Institute.
Garet, M., Porter, A., Desimone, L., Birman, B., & Yoon, K. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 914-945.
Hecht, L. W. (1978). Measuring student behavior during group instruction. Journal of Educational Research, 78(5), 283-290.
Ingvarson, L., Meiers, M., & Beavis, A. (2005). Factors affecting the impact of professional development programs on teachers' knowledge, practice, student outcomes & efficacy. Education Policy Analysis Archives, 13(10), 1-28.
Kazemi, E., & Franke, M. (2003). Using student work to support professional development in elementary mathematics: A CTP working paper. Seattle, WA: Center for the Study of Teaching and Policy.
Leahy, S., Lyon, C., Thompson, M., & Wiliam, D. (2005). Classroom assessment that keeps learning on track minute-by-minute, day-by-day. Educational Leadership, 63(3), 18-24.
Mayer, D. P. (1999, Spring). Measuring instructional practice: Can policymakers trust survey data? Educational Evaluation and Policy Analysis, 21(1), 29-45.
McLaughlin, M., & Talbert, J. (1993). Contexts that matter for teaching and learning: Strategic opportunities for meeting the nation's educational goals. Palo Alto, CA: Stanford University Center for Research on the Context of Secondary School Teaching.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks, CA: Sage.
Mullens, J., & Gayler, K. (1999). Measuring classroom instructional processes: Using survey and case study field test results to improve item construction. Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics.
Mullens, J., & Kasprzyk, D. (1996). Using qualitative methods to validate quantitative survey instruments: Proceedings of the section on survey research methods. Paper presented at the annual meeting of the American Statistical Association, Alexandria, VA.
Mullens, J., & Kasprzyk, D. (1999). Validating item responses on self-report teacher surveys: NCES working paper. Washington, DC: U.S. Department of Education.
Popham, W. J. (2008). Transformative assessment. Alexandria, VA: Association for Supervision and Curriculum Development.
Porter, A. C., Kirst, M. W., Osthoff, E. J., Smithson, J. S., & Schneider, S. A. (1993). Reform up close: An analysis of high school mathematics and science classrooms (Final report to the National Science Foundation on Grant No. SPA-8953446 to the Consortium for Policy Research in Education). Madison, WI: University of Wisconsin-Madison, Wisconsin Center for Education Research.
Putnam, R., & Borko, H. (2000). What do new views of knowledge and thinking have to say about research on teacher learning? Educational Researcher, 29(1), 4-15.
Smith, J. P. (1996). Efficacy and teaching mathematics by telling: A challenge for reform. Journal for Research in Mathematics Education, 27(4), 387-402.
Spillane, J. P., & Thompson, C. L. (1997). Reconstructing conceptions of local capacity: The local education agency's capacity for ambitious instructional reform. Educational Evaluation and Policy Analysis, 19(2), 185-203.
Spillane, J. P., & Zeuli, J. S. (1999). Reform and mathematics teaching: Exploring patterns of practice in the context of national and state reforms. Educational Evaluation and Policy Analysis, 21 (1), 1-27.
Stiggins, R. (2004). New assessment beliefs for a new school mission. Phi Delta Kappan, 86(1), 22-27.
Thompson, M., & Wiliam, D. (2007). Tight but loose: A conceptual framework for scaling up school reforms. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL.
Tocci, C., & Taylor, G. (2007). A state-sponsored pilot project in selected schools in Vermont. Paper presented at the Annual Meeting of the American Educational Research Association and the National Council on Measurement in Education, Chicago, IL.
Wiliam, D. (2007). Keeping learning on track: Classroom assessment and the regulation of learning. In F. K. Lester, Jr. (Ed.), Second handbook of mathematics teaching and learning (pp. 1053-1089). Greenwich, CT: Information Age.
Wilson, S. M., & Berne, J. (1999). Teacher learning and the acquisition of professional knowledge: An examination of research on contemporary professional development. In A. Iran-Nejad & P. D. Pearson (Eds.), Review of research in education (pp. 173-209). Washington, DC: American Educational Research Association.
Wylie, C., Lyon, C., Ellsworth, J., & Martinez, C. (2007). How formative assessment ideas translate from talk to practice. Paper presented at the Annual Meeting of the American Educational Research Association and the National Council on Measurement
Wylie, E. C., Thompson, M., & Wiliam, D. (2007). Keeping learning on track: Impact on student learning. Paper presented at the Annual Meeting of the American Educational Research Association and the National Council on Measurement in Education, Chicago, IL.
Toni A. Sondergeld is a professor in the School of Educational Foundations, Leadership and Policy at Bowling Green State University in Bowling Green, OH. She currently teaches master's level courses in educational assessment, research, and statistics. Dr. Sondergeld's research focuses on program evaluation practices, high stakes testing, survey research methods, and using data for decision-making purposes at all levels.
Courtney A. Bell is a Research Scientist in the Center for Understanding Teaching Quality at ETS in Princeton, NJ. Dr. Bell's work looks across actors in the educational system to better understand the intersections of policy and practice. Her current studies use mixed methods to analyze teacher learning, the measurement of teaching, and the effects of racially desegregated schools on student learning.
Dawn Marie Leusner is a Research Associate in the Learning & Teaching Research Center at ETS in Princeton, NJ. Currently, her research centers around understanding teaching quality, including the specialized knowledge needed for teaching. She also continues contributing to the research around the use of formative assessment as a mechanism for improving teaching and learning in K-12 classrooms.
Toni A. Sondergeld
Bowling Green State University
Bowling Green, OH
Courtney A. Bell
Educational Testing Service
Dawn Marie Leusner
Educational Testing Service
Table 1 Sample Assessment for Learning Techniques Whiteboards Teacher asks a question to the entire class. Each student writes their response on a small dry erase board and holds them up simultaneously for the teacher to assess. Teacher uses this information to assess student understanding and moves on with the lesson accordingly, elaborates/expands upon the content, posing additional questions if warranted or prompting specific students, based on their answer, to explain their thinking to the class. Two Stars and a Wish Teacher structures comments to students by writing two positive things (stars) about their work and one thing that he/she "wishes" the students would do to make it stronger. The "wish" is designed to move learning forward by helping the student focus on an aspect of their work that needs improvement. No Hands Up: Popsicle[R] Sticks Teacher places students' names on Popsicle[R] sticks or tongue depressors. He/she asks a question, waits an appropriate amount of time, and then randomly pulls a stick or sticks for students to answer. Carousel Students send their piece of work around to a small group of students who, in turn, provide constructive feedback. Students have the opportunity to make suggestions for improvement on other students' work using the criteria by which their own products will be judged. Table 2 KLT Reported Impacts in Frequency and Percentage (N=65) Main Theme Supporting Evidence Frequency (Percentage) Support System 43 (66%) Support of Practice 39 (60%) Personal Support 24 (37%) Student Practice 53 (82%) Participation 45 (69%) Student Accountability 26 (40%) Perceived Fairness 13 (20%) Teacher Practice 60 (92%) Formative Assessment Use 54 (60%) Classroom Management 22 (34%) Note. Percentages do not total to 100% and frequencies do not equal 65 as many teachers reported more than one impact of the KLT Program. All percentages are based on the entire sample (N=65).
|Printer friendly Cite/link Email Feedback|
|Author:||Sondergeld, Toni A.; Bell, Courtney A.; Leusner, Dawn Marie|
|Publication:||Teaching and Learning|
|Date:||Mar 22, 2010|
|Previous Article:||Self-study through collaborative conference protocol: studying self through the eyes of colleagues.|
|Next Article:||The dark side of technology: a textual interpretation of school organizations.|