The development and use of a structured teacher observation scale to assess differentiated best practice.

The purpose of this article is to share validation data on a new scale--to judge the efficacy of teacher behaviors in respect to key differentiation strategies found to be effective with gifted learners. The instrument described in the article, the Classroom Observation Scale-Revised (COS-R), has been found to be useful in a number of settings, and for different research and evaluation projects. It began as a dichotomous scale with 40 behaviors being judged as present or absent in classrooms. It has now evolved into a 25-item instrument that judges the efficacy of teacher behaviors using a 3-point scale to assess effective teaching.

The article is organized into several sections. First, an overview of relevant literature is presented, tying the general reform literature on teacher behaviors to the gifted-education literature and documenting the research base for the scale categories. Secondly, the article describes how the scale was developed, outlines its uses, and presents the scale categories, derived subject matter indicators, and the rubric for judging effectiveness. Thirdly, the article documents procedures for using the scale in classroom settings, scoring issues, and technical adequacy data. Finally, the article shows the implications for using the scale in schools for multiple purposes.

A Review of the Literature

Over the last several years, there has been considerable evidence from different studies suggesting that how teachers behave in the classroom, and the instructional approaches they employ, significantly affect the degree to which students learn. Sanders and Rivers (1996) have reported the effects of ineffective teachers over 3 years as depressing student achievement in math by as much as 54%, regardless of the ability of the learner. Wenglinsky (2000) found positive effects from using key practices such as critical thinking and metacognition on student learning in math and science across elementary- and middle-school levels. Several studies on enhancing student learning substantiate emphasis on teaching to high-level concepts for transfer, use of concept mapping to aid connections, and use of metacognition to enhance student learning (National Research Council, [NRC], 2000). These studies have strongly informed our understanding of what strategies teachers need to employ to elevate learning outcomes.

The literature in gifted education suggests that a teacher's use of differentiation strategies is the link to differentiated programs and services for this special population (Renzuli, 2003; VanTassel-Baska, 2003). Yet, replicated studies have suggested that regular classrooms offer very limited differentiated activities (Westberg, Archambault, Dobyns, & Salvin, 1993; Westberg & Daoust, 2003). Moreover, there is no evidence in the literature that teacher strategies in classrooms working with gifted learners are monitored systematically to assess implementation of best practice. Thus, more research on teacher practices is needed in differentiation, and a valid and reliable classroom observation scale would support such research efforts.

All aspects of the reform agenda for schools depend on classroom changes in teacher behaviors to promote student learning at higher levels. Classroom reform, in turn, is highly dependent on positive teacher behavioral change in key areas. Several studies in the reform literature offer insights into ways to impact teachers. A strong background in both content and pedagogy has been found to be indispensable for producing positive teaching behaviors and student-learning outcomes (Joftus & Maddox-Dolan, 2002). A study of math and science programs found that teachers will use strategies linked to content that show results with students (Kennedy, 1999), rather than isolated strategies not embedded in content. Collegiality and support have also been found to be necessary for teacher behavioral change to occur (Garet, Porter, Desimone, Birman, & Yoon, 2001). Another study showed that the use of higher-level reform behavior takes 2 or more years of intensive training to demonstrate results (Borko, Mayfield, Marion, Flexer, & Cumbo, 1993). Based on these findings, it is clear that attention to classroom-level instruction must be carefully monitored to know the improved nature of teaching and learning occurring at the classroom level.

Yet, use of teacher evaluation alone or disconnected from on-going professional development appears to be unsuccessful as a catalyst for teacher change. A recent study of teacher evaluation practices in three districts in Wisconsin (Kimball, 2002) found that few teachers reported substantial changes in their instructional practice as a result of their evaluation experiences, and the large majority of teachers did not see the evaluation process as an incentive to seek out professional development opportunities.

A component linking teacher effectiveness to student outcomes is a necessary part of any effective educational evaluation system (Sanders & Horn, 1998). Vandevoort, Amrein-Beardsley, and Berliner (2004) examined academic performance results of elementary students in both National Board Certified Teachers and those who were not certified for 4 years, finding that students in the classes of National Board Certified Teachers surpassed students in the non-Board certified teachers in almost three quarters of the comparisons, suggesting that teacher use of research-based strategies impact student learning.

Gifted-education practice has had an uneasy alliance with key facets of educational reform (VanTassel-Baska, 1993) including standards, assessments, grouping patterns, and materials, supporting the need for challenging standards but questioning the levels at which the standards are addressed in classrooms. To its credit, the field is recognized for advancing the introduction of innovative instructional practices into the classroom, such as inquiry learning, critical and creative thinking skills, higher-order questioning strategies, metacognition, and the use of rich and varied curricular materials, rather than sole reliance on textbooks (Tomlinson & Callahan, 1992). Most recently, the introduction of content-based curriculum tied to state and national standards and evaluated on student learning gains (VanTassel-Baska, Bass, Ries, Poland, & Avery, 1998; VanTassel-Baska, Zuo, Avery, & Little, 2002) has limited the field to curriculum reform, with the emphasis on using the standards as a basis for instruction but extending learning opportunities beyond the stated standard to achieve instructional impact.

Other studies showed that effective teachers of the gifted incorporated a variety of materials and strategies in the instructional delivery process (Ford & Trotman, 2001 ; Hansen & Feldhusen, 1994; Nelson & Prindle, 1992; Story, 1985; Westberg & Archambault, 1997); differentiated instruction was particularly cited as an approach that meets the needs of gifted and talented learners (Archambault et al.,1993; Coleman & J. J. Gallagher, 1995; J. J. Gallagher, 1985; Jellen & Verduin, 1986; Maker, 1982; Renzulli & Reis, 1998; Tomlinson, 1999; VanTassel-Baska, et al., 2002; Ward, 1980).

Yet there is little evidence that gifted-education programs systematically assess student gains using learning measures that go beyond state-assessment tests (Avery & VanTassel-Baska, 2002). In the absence of authentic student impact data, evaluators must often rely on the quality of the instructional experience as a proxy method for investigating program effectiveness. In fact, teacher effectiveness has been shown to be the main determinant of student progress (Sanders & Horn, 1998). In meeting the needs of the gifted learner, the observer must focus on three dimensions of practice: good teaching in general, key elements of educational reform, and differentiation for high-ability learners.

There are certain assumptions made about the teaching act that underlay the use of any observation tool, especially one like the COS-R (VanTassel-Baska et al., 2003) which is extensive and in-depth with respect to behaviors. Teaching is a complex social activity requiring the capacity to split attention by student, by area of the room, and by activity. It requires making multiple decisions during a teaching episode by instructional regulation, by strategies, use of time, and lesson emphasis. Moreover, teachers must think in complex ways in order to implement many behaviors simultaneously. For example, one type of thinking that teachers must employ relates to following the lesson plan, responding to pacing, modulating student-teacher interaction, and ensuring a variety of stimuli. A second type of thinking required of teachers is awareness of teaching to multiple objectives, such as teaching a concept at the same time they are teaching content and skills, while attending to the teaching of group dynamics. A third way of thinking involves orchestrating feedback strategies and responses, flexibility in grouping patterns, and questioning and activity choices. Planning, monitoring, and assessing both group and individual learning is another level of thinking that teachers must employ in the moment of teaching (in medias res). Finally, teachers need to think about sequencing their lessons. How does the current lesson relate to yesterday's and how will it link to tomorrow's?

Teaching has traditionally been a solitary activity; thus, teachers are not socialized to having external observations of their work, and oftentimes are uncomfortable with the observation process. Yet, improvement in teaching clearly requires a change in teacher behaviors that promote learning in students. Such improvement appears to imply the use of higher order thinking, problem solving, and metacognitive approaches. In order to ensure that teachers are employing such strategies, some form of monitoring teacher behaviors must occur.

Goals of Classroom Observation

Evidence of classroom practices being used is a seminal part of education reform because it provides the database on which future work with teachers can proceed (Guskey, 2000; Kennedy, 1999). Thus, classroom observation affords an opportunity to access the actual instructional experience that is at the heart of teaching and learning. It provides a nexus between the input variables of the teacher and his or her students and the process of instruction itself--a process that combines instructional intent (goals and objectives), curriculum resources and materials, instructional strategies, and classroom management skills within a delimited unit of time. It is the one part of professional development that allows the critical pieces of teacher knowledge and skills to come together in an authentic opportunity to gain insight about the quality of the learning experiences that are delivered.

One way of thinking about classroom observation is to see it as a performance-based assessment of the teacher within the context of the learning environment. It affords many of the features of performance-based assessment with the teacher, rather than the student, as the unit of focus. For instance, it is a relatively open-ended experience, with teachers exercising much control over the selection of the lesson to be taught. It allows for the demonstration of complex and higher-order behaviors, recognizing that good teaching derives from a sophisticated set of skills that unfold in an integrated way. It also allows for self-assessment, providing a metacognitive dimension to the experience. Most importantly, by using a structured form, it provides a benchmark against which the teaching process can be assessed, based on expectations derived from best practice in a given field.

However, it is important to distinguish between program and teacher evaluation in terms of the parameters used in the classroom observation process. The focus of program evaluation targets the collective whole. Teacher behavior is sampled through the classroom observation process, and the intent is to allow teachers to prepare for the observation period in order to reduce the level of threat that permeates the current school climate of accountability. By aggregating data across different teachers and classrooms, a snapshot of instructional practice is created that helps inform our understanding of program quality.

Direct observation is also an important component of a teacher-evaluation process, but deeper sampling of the individual being assessed must occur when used for this purpose. Also, teacher evaluation should involve some opportunity for unplanned visits to the classroom to assess the consistency of instructional performance. While the two uses of classroom observation (program and teacher evaluation) can complement one another, it is important to acknowledge that the limited sampling template used for program evaluation is not sufficient to make inferences regarding individual staff performance tied to retention or promotion decisions.

Research on Existing Observation Scales

Extant literature on classroom observation tools in the field of gifted education is limited. Feldhusen and Hoffman (1988) developed an observation scale, for use as part of practicum requirement. The Purdue Observation Form covered 10 areas of expected competencies a teacher of gifted students was expected to have, including subject matter coverage, clarity of teaching, motivational techniques, pace of instruction, opportunities for self-determination of activities by student, student involvement, teacher-student interactions, extended home-work assignment, and teaching aids and materials (p. 38). Westburg, Dobyns, and Archambault (1990) developed the Classroom Practices Record (CPR) for the purpose of documenting types of differentiation activities (e.g., grouping pattern, self-initiated independent study, acceleration, higher cognitive process, questioning strategies, and wait time, cited in Cassady et al., 2004). Cassady et al. developed a more comprehensive scale (Differentiation Classroom Observation Scale) documenting a wider range of differentiation activities embracing cognitive activities, students' level of engagement, and the learning director.

These observation tools have all been used to serve a number of purposes: to measure teacher competencies (Feldhusen & Hoffman, 1988), to document differentiation activities, to examine curricular and instructional practices, and to use program evaluation (Cassady et al., 2004; Westburg et al., 1993). The structure of these observation scales has been more helpful in documenting the presence of a series of activities than examining the effectiveness of teachers' instructional practices.

Acknowledging some similar characteristics with these observation tools, the COS-R is unique in that it has a clear focus on teachers' expected behaviors tied to best practices from both the reform literature and the gifted literature. It has clear evidence of technical adequacy, an important feature of a sound instrument. The COS-R also can be targeted for use in different subject domains and school levels, using domain-specific indicators. Finally, the COS-R was designed to examine the relationship between teachers' instructional effectiveness and students' attentiveness and responsiveness to teachers' instruction, another indicator of measuring instructional effectiveness.

Observation Scale Development

The process of development for the teacher-observation scale used in this study, the COS-R, involved several stages and forms, occurring over more than a decade. At the first stage, the project team reviewed extant literature on effective teaching, differentiated instruction for gifted learners, educational reform and change, and professional development research. Based on expectations derived from best practices in mainstream and gifted-education classrooms, the form was developed to be utilized in all classrooms and in all subject areas. Only the most critical behaviors for general teaching and differentiation features culled from research-based evidence of effective classroom-based instructional behaviors were included. These behaviors focus on the use of strategies that promote student learning and growth, especially in the area of higher order thinking, problem solving, and metacognition. Clusters were created based on predominant teaching behaviors within each category. In its final iteration, the COS-R comprised a total of 25 items in six clusters. Table 1 shows the six categories and specific behavioral items for each.

A description of the distinctive behaviors that characterize effective teachers' classes in each of these clusters follows.

General Teaching Behaviors

Curriculum Planning and Delivery (CPD)

Effective teachers thoroughly plan and organize for instruction (Guskey, 2000; Stronge, 2002). In planning and structuring instruction and activities, teachers have an explicit set of high expectations for student performance, and communicate the importance of learning (Cawelti, 1999). They incorporate activities for students to apply new knowledge (Marzano, Picketing, & McTighe, 1993), engage students in metacognitive processes (Costa, 2001; Wang, Haertel, & Walberg, 1993), and encourage student reflection and expression of thought (Good & Brophy, 1997).

Differentiated Teaching Behaviors

The development of categories and items on the scale were derived from the literature on best practices with gifted learners (See Table 1). Instructional practices in classrooms for the gifted emphasize the importance of concept development, thinking and reasoning, problem solving, and flexible accommodations for working with highly able learners (VanTassel-Baska, 2003). Gifted learners are motivated when engaged in learning basic skills in context rather than in isolation, functioning consistently at high levels of thinking, making connections among disciplines, solving real problems, presenting products to real audiences, and dealing with ambiguities and behaving like professionals in the field (Tomlinson, 1999).

Accommodations for Individual Differences (AID)

Gifted students are not homogeneous. They differ in their abilities, their readiness to learn, their interests, and their styles of learning. Effective teachers recognize student variance and address it appropriately (Tomlinson, 1999). Besides whole-class instruction and discussion, teachers design structured activities and questions to allow students to discover ideas individually. They provide opportunities for individual or group learning and accommodate individual or subgroup differences by allowing choice in material and task selection (VanTassel-Baska, 2003).

Problem Solving (PS)

Gifted students have early capacity for problem solving, and they possess unique characteristics that make them adept at problem finding (J. J. Gallagher, 2001). The three items in this category are derived from the stages of well-established research-based models of problem solving like that of Parnes' and Osborne's model (Isakson, Puccio, & Treffinger, 1993).

Critical Thinking Strategies (CRI)

Effective teachers of the gifted emphasize higher order thinking skills that are integral to all content areas and everyday life experiences (Paul, 1992; Paul & Elder, 2001). Students demonstrate their understanding of advanced content by making generalizations from the concrete to the abstract and vice versa, and synthesizing information within and across disciplines (VanTassel-Baska, 2003; Wenglinsky, 2000).

Creative Thinking Strategies (CRE)

Torrance (1981, cited in Cramond, 2001) found in a study that "teachers who made a difference were those who enabled their students to hold on to their creativity" (p. 405). Effective teachers use an array of strategies to foster creativity. They have students explore diverse view points to reframe ideas. They solicit diverse thoughts about issues from students and encourage them to elaborate on their ideas. Teachers demonstrate open-mindedness and tolerance of imaginative and unusual solutions to problems, and encourage students to do the same (Davis, 2001; Treffinger, 1995).

Research Strategies (RS)

Gifted students should have the opportunity for independent study, be equipped with the prerequisite skills for effective research, and develop these skills to a sophisticated level (Reis & Renzulli, 1992). Students should be required to use appropriate and varied research techniques to gather evidence from multiple sources, interpret, draw inferences, and make conclusions from them. They also should be given the opportunity to communicate their research findings to relevant audiences (VanTassel-Baska & Little, 2003).

Examples of Items in the Clusters

As mentioned earlier, the COS-R is divided into six subscales. The first subscale focuses on curriculum planning and delivery. Examples of observable teacher behaviors for this subscale include setting high expectations for student performance and asking students to reflect on what they have learned. The second subscale focuses on accommodations for individual differences; examples include accommodating individual differences through materials, conferencing, and/or task assignments, and encouraging multiple interpretations. The third subscale focuses on problem solving, specifically the heuristic of brainstorming, problem identification and definition, and developing solutions to problems based on generalizations. The fourth subscale targets critical thinking strategies and includes skills such as engaging students in comparing and contrasting ideas as well as encouraging students to synthesize information within and across disciplines. The fifth subscale focuses on creative thinking strategies; examples include soliciting diverse thoughts about issues or ideas from students, and encouraging students to demonstrate open-mindedness and tolerance of imaginative solutions to problems. Finally, the sixth subscale focuses on research strategies such as gathering evidence from multiple sources, analyzing data, and encouraging students to identify consequences and implications of their findings.

Demographic information about the class observed is captured in the first section of the COS-R. Information about the teacher, as well as the class (grade level, number of boys and girls, ethnicity, classroom desk arrangement, etc.) provides the context for the lesson observation.

Item Development Process

Items were developed to reflect the key behaviors in each category. A review of the relevant literature was used to construct indicators (see Table 2). The minimum number of behaviors per category was three, and the maximum was five. Items were written with the following considerations: All items should be succinct and the use of jargon should be avoided; all items should be written in the active voice to clearly describe the teaching behavior; only instructional behaviors that can be observed in classroom settings should be included; and items should be worded positively.

To emphasize the teacher as the unit of focus, all items begin with the stem "The teacher ..." followed by the description of the key instructional behavior. Similarly, the subject-based descriptions of teaching behaviors are focused on the actions of the teacher. The items in each of the clusters are in Table 1.

Subject-based Indicators

To complement the six clusters of teaching behaviors, sets of domain-specific indicators were developed. The purpose was to provide observers with illustrative examples of observable classroom behaviors as they pertain to different subject areas. This was to reduce variability in the interpretation of individual items within each cluster in any content area, and to enhance inter-rater reliability. For example, for the item "Encouraged students to judge or evaluate situations, problems, or issues" in the cluster of "critical thinking strategies" see Table 3.

The indicators for each subject area were developed for each behavioral item and reviewed by content specialists using content-based curricula tied to state and national standards, and feedback from content- and gifted-education specialists was sought to improve the clarity of the indicators' description and to ensure that prototypical examples were included. The list of behavioral indicators, which usually numbers 3 to 5, is not meant to be exhaustive, merely illustrative. Thus, appropriate behaviors in each category that are not on the list should also be rated.

Rating Scale

Each item on the scale is rated for its level of effectiveness. There are three levels on this rating scale with 3 being effective, 2 being somewhat effective, and I being ineffective; a rubric description of each level is included. A behavior is rated as effective if the teacher evidenced careful planning and classroom flexibility in implementation of the behavior, eliciting many appropriate student responses; the teacher was clear, and sustained focus on the purposes of learning. A behavior is rated as somewhat effective if the teacher evidenced some planning and/or classroom flexibility in implementation of the behavior, eliciting some appropriate student responses. The teacher was sometimes clear and focused on the purposes of learning. An ineffective rating is given if the teacher evidenced little or no planning and/or classroom flexibility in implementation of the behavior, eliciting minimal appropriate student responses, and the teacher was unclear and unfocused. However, if a behavior item in the COS-R was not observed, no rating is to be given, and a N/O or not observed note is to be documented.

Effectiveness Rating Issues

At the training sessions conducted to enhance inter-rater reliability, several questions arose regarding the use of the scale. For example, should a teacher be rated a 3 even if the behavior is not truly exemplary? Or, how many indicators need to be observed in order to score the category at a 2 or 3 level? Based on these discussions, a set of guidelines was developed to clarify the use of the scale. Observers should employ the following guidelines when rating the effectiveness of any given teaching behavior:

1. An item checked N/O was not considered a negative rating and did not address teacher effectiveness in any way.

2. An item checked 1 was negative whereas an item that was checked N/O was neutral.

3. Only one or two of the indicators for each item in a subscale needed to be present to warrant a rating of 2 or even 3.

4. A rating of 3 (effective) was not to be judged as equivalent to exemplary and therefore should not be withheld on the grounds that the observed behavior was not outstanding, even though it is the top rating on the scale.

5. In order for a behavior to be rated, it had to be deliberate and sustained during the period of the observation.

6. Appropriate behaviors that are not listed in the descriptors should be acknowledged through the comments section.

Piloting

The original version of the COS-R was the Classroom Observation Form (COF) which had 40 items in nine subcategories. No subcategory had fewer than 3 items, while the number of items in the largest subcategory was 9. The COF was piloted with 50 teachers teaching in the Saturday Enrichment Program at the College of William and Mary. Based on the data gathered from the pilot, items in each of the subcategories were reviewed for their contributions to the overall reliability (i.e., internal consistency) of the instrument, while ensuring that they were representative of key teaching behaviors. To reduce redundancy according to the rule of parsimony, and to increase the rigor of the scale, the number of subcategories and items were reduced. The final version of the COF, renamed the COS-R has 25 items in six subcategories. Criteria employed to make such reductions were: internal consistency, keeping items and categories discrete, and enhancing ease of use.

Data using the COS-R were collected in two waves of observation in the first 2 years of implementation of Project Athena, a research study funded by a U.S. Department of Education Javits grant to explore the effects of language arts curriculum treatment on students' reading achievement and critical thinking and on teaching practices in grades 3, 4, and 5. The COS-R was also used in a replication study with a group of teachers in an enrichment program at the College of William and Mary. Open-ended scripting of lessons, completion of the scale, using the subject-based indicators, and both individual and collaborative ratings were employed in all phases of the data collection.

Implementation and Scoring

Observers do a narrative recording of instructional behaviors observed during the lesson. To be included in the script are a description of teacher's actions, time spent on activities and transitions between activities, and a record of teachers' questions and student responses as well as the pattern of interaction. Observers are also encouraged to note questions they might have about the lesson.

Observers script their observations independently, and then code them according to the 25 items in the six categories. Observations are typically based on a 30- to 50-minute lesson. Each observer is to determine first if a behavior is observed or not observed. If it is not observed, the N/O column should be checked. If a behavior is observed, the observer is to rate its effectiveness using the 3-point scale provided. Researchers encourage the use of two observers in each classroom who confer after they have done individual ratings in respect to how they have rated individual teachers.

Observers are strongly encouraged to cite evidence in the lesson script to support their ratings, and to achieve consensus using script-based evidence. Scores for each individual item as well as each subscale can be computed to provide subscale means and item means to provide a snapshot of instructional practice within and across classrooms.

Technical Adequacy

The technical characteristics of the COS-R were investigated during two waves of data collection during the first 2 years of the implementation of Project Athena.

Reliability

Reliability is the degree to which measures are free from error and therefore yield consistent results. There are three major sources of error: factors in the measure itself (poorly written items), scoring factors (clarity of scoring rubrics), and factors in the people answering the items on the measure (Rudner & Schafer, 2001). Internal consistency refers to the extent to which all questions or items assess the same characteristic, skill, or quality. Wasserman and Bracken (2003) suggest that scales intended for research applications should be minimally reliable at a level of .70, and preferably .80. This recommended level guided the selection of items for the final version of the COS-R.

As noted earlier, the longer COF scale was piloted on a group of 50 teachers teaching in the Saturday Enrichment Program for gifted children at the College of William and Mary. Based on the internal consistency reliability analyses of the pilot data, the number of categories was reduced to six, and the number of items was reduced to 25.

The COS-R was used twice in Project Athena Classroom Observation during Fall 2003 and Spring 2004. Twenty-three teams of observers visited 73 classrooms in two rounds of observations. Two waves of data were collected from these observations. Reliability analyses were conducted for the overall scale as well as for each of the six subscales. Table 4 presents reliability results from both piloting and implementation data.

The analyses of the two implementation observation periods showed that overall, the scale was highly reliable (Alpha = .91 to .93). For both observations, the subscale reliability for all of the clusters averaged above .70. These high reliability coefficients across both observations attest to the reliability of the items on the instrument.

Inter-rater Reliability

Inter-rater reliability ensures that there is consistency in the use of a rating scale, regardless of who the trained observer is in the classroom. In order to assess inter-rater reliability, the research team had each observer complete an individual rating for each observation in the pilot and then analyzed the correlation between each pair of ratings. The inter-rater reliability of the COS-R reached .87 and .89, respectively, across each observation period.

To enhance consistency of interpretation further, each potential user of the form receives a half-day training on the form. This training consists of an introduction to the COS-R scale, observation of videotaped lessons (which participants script and rate), a discussion of ratings, and a second videotape practice session followed by discussion.

Content Validity 

The COS-R was sent to six experts in gifted education to review its content validity. On a prescribed form developed for this purpose, reviewers were asked to rate the COS-R on two dimensions:

1. The importance of each behavioral item on the scale, with respect to its relevance to the category and its importance as an indicator of gifted best practice: Raters used a 3-point scale to judge the level of importance, with 3 being very important and 1 being not important.

2. The accuracy of the language used to describe the behaviors: The clarity of the language was judged on a 3-point scale, with 3 being very clear, and 1 designating a lack of clarity in the language used to describe the behavior.

Three professors and scholars in gifted education and three practitioners and administrators in local school districts were identified for the content-validity process. Four of them returned the prescribed forms with their ratings and comments. Based on the agreement of these four experts, the intraclass coefficient Alpha was used to assess rater agreement on the scale. Their agreement on the two dimensions of the validity was .86 for the importance of each item, and .99 for the clarity of language. These results are presented in Table 5.

The reliability study of the COS-R was replicated in Spring 2004 with 17 teachers in a Saturday Enrichment Program for gifted students at the College of William and Mary. Of the 17 classes taught by the participating teachers, 5 were science-related classes, 5 were humanities-related, 3 were math-related, and 2 focused on developing problem-solving skills. Two observers who had been trained and had experience using the COS-R observed the teachers in classes spanning grades pre-K to Grade 9, The inter-rater reliability for this replication study was .92. The subscale reliability for two of the clusters, "curriculum planning and delivery" (Alpha = .91) and "creative thinking skills" (Alpha = .89) were especially strong.

Scale Development Limitations

Additional analyses on the COS-R have not yet been done, although preliminary factor analyses have been run, suggesting the need to reorganize clusters of items. Further work in this area is needed. Moreover, predictive validity studies need to be run to ascertain the link between teaching behaviors and student learning outcomes. These studies are planned after Year 3 student outcome data for Project Athena are available.

The COS-R is being used currently in a study of secondary classrooms in five different subject areas and cross-cultural settings. It is also being used in Project Athena across 3 years in schools to judge positive teacher changes in differentiated behaviors over time. Data across 2 years suggest that experimental teachers significantly out perform control teachers on all aspects of the form and continue to show growth across the 2-year span of observation.

Implications for the Use of the COS-R

If classroom practice for the high-ability learner requires the use of "differentiated" strategies, then how is this requirement translated into expectations for teacher performance and accountability? What constitutes "practice" in classrooms serving gifted learners? The COS-R provides a mechanism for using quantitative parameters to answer these questions. We cannot afford to ignore teacher behaviors that theory and research have found contribute to improved teaching effectiveness. The COS-R is one way to assess individual teacher performance in response to high-ability learners. Moreover, it may provide a direct link to understanding gifted-student outcomes as well. Another corresponding scale is under development to calibrate student behaviors that resulted from the suggested teacher behaviors in the scale. Coupled with student pre-post gain data, we then can assess the influence of teaching on gifted-student learning outcomes.

Research on the use of an earlier version of the COS (i.e., COF) suggests that while teachers of the gifted appear strong in many categories of good teaching, they fall short in many others that examine differentiation practices (VanTassel-Baska, 2004). For instance, higher-level-thinking models were used less than might be expected in classrooms when gifted learners form a critical mass of the student body. Very little emphasis was placed on accelerative strategies in gifted programs. Rate differences between gifted learners were not routinely attended to by classroom teachers, even those trained in gifted education. Use of models that explore different types of thinking and problem-solving behavior were not routinely employed in gifted-program classrooms. The most prevalent problem-solving behavior observed was brainstorming, and many times this was used in isolation of more fulsome models that examined a broader array of problem-solving behavior. Little evidence of the use of metacognition in gifted classrooms was observed. Moreover, little emphasis on out-of-class learning was found in observed teachers of the gifted classrooms as judged by their lack of provision for extension activities. Individualizing for differences within the gifted population, critical thinking, and problem solving were found to be underutilized in most observed classrooms. Core teaching behaviors like lesson planning and clarity in providing directions were more routinely observed. Curriculum reform elements were in evidence unevenly, with greater prevalence of concept teaching than other behaviors.

Thus, the COS-R provides direct evidence of the need for specific emphases on differentiation strategies in professional development programs and may be effectively employed as a planning tool for that purpose. Professional development plans must take into account the evolving skills of teachers such that workshops and seminars are more tightly focused on needed skills as opposed to desired areas of interests. Gifted programs need to consider the sophistication of using higher-level skills effectively and thus the need for teachers to develop these skills over time through appropriate methods.

The COS-R may also prove useful in charting overall growth in desirable teacher behaviors by school or district as has been found in other checklists with self-monitoring capabilities (Hague & Walker, 1996). Diagnoses of individual teacher skills needed to work effectively with gifted learners can also easily occur through the use of the COS-R. Videotape analysis of teaching can employ a self-assessment component for teachers as well as a component for external observers.

Conclusion

Teaching is a performance art. Therefore, the test of success is demonstrating mastery of relevant techniques. Use of monitoring approaches including coaching, structured observation, and videotape analysis all increase the chance that teachers will improve in important ways in behaviors that optimize learning for our best students. If "differentiation" is to work in our schools, we need teachers prepared to demonstrate efficacy in these core areas.

Manuscript submitted November 17, 2005.

Revision accepted January 16, 2006.

REFERENCES

Archambault, F. X., Westberg, K. L., Brown, S. W., Hallmark, B. W., Zhang, W., & Emmons, C. (1993). Classroom practices used with gifted third and fourth grade students. Journal for the Education of the Gifted, 16, 103-119.

Avery, L. D., & VanTassel-Baska, J. (2002). The impact of gifted education evaluation at state and local levels: Translating results into action. Journal for the Education of the Gifted 25, 153-176.

Borko, H., Mayfield, V., Marion, S., Flexer, R., & Cumbo, K. (1993). Teachers developing ideas and practices about mathematics performance assessment: Successes, stumbling blocks, and implications for professional development. Teaching and Teacher Education, 13, 259-278.

Cassady, J. C., Neumeister, K. L. S., Adams, C. M., Cross, T. L., Dixon, F. A., & Pierce, R. L. (2004). The differentiated classroom observation scale. Roeper Review, 26, 139-146.

Cawelti, G. (Ed.). (1999). Handbook of research on improving student achievement: The schooling practices that matter most (2nd ed.). Arlington, VA: Educational Research Service.

Coleman, M. R., & Gallagher, J. J. (1995, September/October). Appropriate differentiated services: Guides for best practices in the education of gifted children. Gifted Child Today, 18, 32-33.

Costa, A. L. (Ed.). (2001). Developing minds: A resource book for teaching thinking (3rd ed.). Alexandria, VA: Association for Supervision and Curriculum Development.

Cramond, B. (2001). Fostering creative thinking. In F. A. Karnes & S. M. Bean (Eds.), Methods and Materials for Teaching the Gifted (pp. 399-444). Waco, TX: Prufrock Press.

Davis, G. A. (2001). Identifying creative students, teaching for creative growth. In N. Colangelo & G. A. Davis (Eds.), Handbook of gifted education (2nd ed.). Boston: Allyn & Bacon.

Dods, R. F. (1997). An action research study of the effectiveness of problem-based learning in promoting the acquisition and retention of knowledge. Journal for the Education of the Gifted, 20, 423-437.

Feldhusen, J., & Huffman, L. (1988). Practicum experiences in an education program for teachers of the gifted. Journal for the Education of the Gifted, 12, 34-45.

Ford, D. Y., & Trotman, M. F. (2001). Teachers of gifted students: Suggested multicultural characteristics and competencies. Roeper Review, 23, 235-239.

Gallagher, J. J. (1985). Teaching the gifted child (3rd ed.). Boston: Allyn & Bacon.

Gallagher, J. J. (2001). Personnel preparation and secondary education programs for gifted students. Journal of Secondary Gifted Education, 12, 133-138.

Gallagher, S. A., Stepien, W. J., & Rosenthal, H. (1992). The effects of problem-based learning on problem solving. Gifted Child Quarterly, 36, 195-200.

Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective: Results from a national sample of teachers. American Research Journal, 38, 915-945.

Good, T. L., & Brophy, J. E. (1997). Looking in classrooms (7th ed.). New York: Longman.

Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press.

Hague, S. A., & Walker, C. (1996). Creating powerful learning opportunities for all children: The development and use of a self-monitoring checklist for teachers. U.S. Department of Education, Office of Educational Research and Improvement, Education Resource Information Center. (ERIC Document Reproduction Service No. ED396844)

Hansen, J. B., & Feldhusen, I. F. (1994). Comparison of trained and untrained teachers of gifted students. Gifted Child Quarterly, 32, 216-221.

Isakson, S. G., Puccio, G. J., & Treffinger, D. J. (1993). An ecological approach top creativity research: Profiling for creative problem solving. Journal of Creative Behavior, 27, 149-170.

Jellen, H. G., & Verduin, J. R. (1986). Handbook for differential education of the gifted: A taxonomy of 32 key concepts. Carbondale: Southern Illinois University Press.

Joftus, S., & Maddox-Dolan, B. (2002). New-teacher excellence: Retaining our best. Alliance for excellent education. Retrieved October 11, 2006, from http://www.all4ed.org/publications/NewTeacherExcellence/NTE.pdf

Kennedy, M. (1999). Form and substance in mathematics and science professional development. NISE Brief 3(2), 1-7.

Kimball, S. M. (2002). Analysis of feedback, enabling conditions, and fairness perceptions. Madison: University of Wisconsin-Madison, Wisconsin Center for Education Research, Consortium for Policy Research in Education.

Kulik, J. A., & Kulik, C.--L. C. (1992). Meta-analytic findings on grouping programs. Gifted Child Quarterly, 36, 73-77.

Little, C., Feng, A., VanTassel-Baska, J., Rogers, K., & Avery, L. (2002). Project Phoenix: Final report on social studies curriculum effectiveness. Williamsburg, VA: The College of William and Mary, Center for Gifted Education.

Maker, C. J. (1982). Curriculum development for the gifted. Austin, TX: PRO-ED.

Marzano, R. J. (1998). A theory-based meta-analysis of research on instruction. Aurora, CO: Mid-continent Research for Education and Learning.

Marzano, R. J., Gaddy, B. B., & Dean, C. (2000). What works in classroom instruction. Aurora, CO: Mid-continent Research for Education and Learning.

Marzano, R. J., Picketing, D., & McTighe, J. (1993). Assessing student outcomes (Vols. 1-5). Alexandria, VA: Association for Supervision and Curriculum Development.

National Research Council. (2000). How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press.

Nelson, K. C., & Prindle., N. (1992). Gifted teacher competencies: Ratings by rural principals and teachers compared. Journal for the Education of the Gifted, 15, 357-369.

Paul, R. (1992). Critical thinking: What every person needs to survive in a rapidly changing world. Rohnert Park, CA: Sonoma State University, Center for Critical Thinking and Moral Critique.

Paul, R., & Elder, L. (2001). Critical thinking: Nine strategies for everyday life: Part I. Journal of Developmental Education, 24(1), 40-41.

Reis, S. M., & Renzulli, J. S. (1992). The library media specialist's role in teaching independent study skills to high ability students. Library Media Quarterly, 21(1), 27-35.

Renzulli, J. S. (2003). Conception of giftedness and its relationship to the development of social capital. In N. Colangelo & G. A. Davis (Eds.), Handbook of gifted education (3rd ed.). Boston: Allyn & Bacon.

Renzulli, J. S., & Reis, S. M. (1998). Talent development through curriculum differentiation. NASSP Bulletin, 82(595), 61-64.

Rogers, K. (2002). Re-forming gifted education: Matching the program to the child. Scottsdale, AZ: Great Potential Press.

Rudner, L. M., & Schafer, W. D. (2001). Reliability. ERIC Digest. College Park, MD: ERIC Clearinghouse on Assessment and Evaluation. (Eric Document Reproduction Services No. ED 458213)

Sanders, W. L., & Horn, S. P. (1998). Research findings from the Tennessee value-added assessment system (TVASS) database: Implications for educational research and evaluation. Journal of Personnel Evaluation in Education, 12, 247-256.

Sanders, W. L., & Rivers, J. C. (1996). Cumulative and residual effects of teachers on future students' academic achievement. Knoxville: University of Tennessee, Value-Added Research and Assessment Center.

Story, C. M. (1985). Facilitator of learning: A micro-ethnographic study of the teachers of the gifted. Gifted Child Quarterly, 29, 155-159.

Stronge, J. H. (2002). Qualities of effective teachers. Alexandria, VA: Association for Supervision and Curriculum Development.

Tomlinson, C. A. (1999). The differentiated classroom: Responding to the needs of all learners. Alexandria, VA: Association for Supervision and Curriculum Development.

Tomlinson, C. A., & Callahan, C. M. (1992). Contributions of gifted education to general education in a time of change. Gifted Child Quarterly, 36, 183- 189.

Treffinger, D. (1995). Creativity, creative thinking, and critical thinking. In search of definitions. Sarasota, FL: Center for Creative Thinking.

Treffinger, D. J., & Isaksen, S. G. (2005). Creative problem solving: The history, development, and implications for gifted education and talent development. Gifted Child Quarterly, 49, 342-353.

Vandevoort, L., Amrein-Beardsley, A., & Berliner, D. (2004, September). National board certified teachers and their students' achievement. Education Policy Analysis Archives, 12(46). Retrieved August 28, 2005, from http://epaa.asu.edu/epaa/v12n46

VanTassel-Baska, J. (1993, July-August). Linking curriculum development for the gifted to school reform and restructuring. Gifted Child Today, 16(5), 34-37.

VanTassel-Baska, J. (2000, August). Standards of learning and goodness of fit. Tempo, TX: Association for the Gifted.

VanTassel-Baska, J. (2003). Curriculum planning and instructional design for gifted learners. Denver, CO: Love Publishing.

VanTassel-Baska, J. (2004). Assessing classroom practice: The use of a structured observation form. In J. VanTassel-Baska & A. Feng (Eds.), Designing and utilizing evaluation for gifted program improvement (pp. 87-108). Waco, TX: Prufrock Press.

VanTassel-Baska, J., Avery, L., Struck, J., Feng, A., Bracken, B., Drummond, D., et al. (2003). The William and Mary classroom observation scales revised (COS-R). Williamsburg, VA: Center for Gifted Education.

VanTassel-Baska, J., Bass, G., Ries, R., Poland, D., & Avery, L. D. (1998). A national pilot study of science curriculum effectiveness for high ability learners. Gifted Child Quarterly, 42, 200-211.

VanTassel-Baska, J., & Little, C. (Eds.). (2003). Content based curriculum for high ability learners. Waco, TX: Prufrock Press.

VanTassel-Baska, J., Quek., C., & Feng, A. (2005). The classroom observation scale-revised manual. Williamsburg, VA: The College of William and Mary, Center for Gifted Education.

VanTassel-Baska, J., & Stambaugh, T. (2005). Challenges and possibilities for serving gifted learners in the regular classroom. Theory Into Practice, 44, 211-217.

VanTassel-Baska, J., Zuo, L., Avery, L. D, & Little, C.A. (2002). A curriculum study of gifted student learning in the language arts. Gifted Child Quarterly, 46, 30-44.

Wang, M. C., Haertel, G. D., & Walberg, H. J. (1993). What helps students learn? Educational Leadership, 51, 74-79.

Ward, V. S. (1980). Differential education for the gifted. Columbus, OH: Merrill.

Wasserman, J. D., & Bracken, B. A. (2003). Psychometric considerations of assessment procedures. In J. Graham & J. Naglieri (Eds.), Handbook of assessment psychology (pp. 43-66). New York: Wiley.

Wenglinsky, H. (2000). How teaching matters. Princeton, NJ: Educational Testing Service.

Westberg, K. L., & Archambault, F. (1997). A multi-site case study of successful classroom practices for high ability students. Gifted Child Quarterly, 41, 42-51.

Westberg, K. L., Archambault, F., Dobyns, S. M., & Salvin, T. (1993). The classroom practices observation study. Journal for the Education of the Gifted, 16, 120-146.

Westherg, K. L., & Daoust, M. E. (2003, Fall). The results of the replication of the classroom practices survey replication in two states. The National Research Center on Gifted and Talented Newsletter, 3-8.

Westherg, K. L., Dobyns, S. M., & Archambault, F. (1990). The training manual for The Classroom Practices Record. Storrs: The University of Connecticut, The National Research Center on the Gifted and Talented.

Joyce VanTassel-Baska is the Jody and Layton Smith Professor of Education at the College of William and Mary in Virginia, where she has developed a graduate program and a research and development center in gifted education. Dr. VanTassel-Baska has published widely, including five recent books and over 300 monographs, book chapters, and articles in refereed journals. Dr. VanTassel-Baska is the former chief editor of Gifted and Talented International and is the current president of NAGC. E-mail: jlvant @wm.edu

Chwee Quek is the gifted specialist of gifted education at the Gifted Education Branch, the Ministry of Education, Singapore. Dr. Quek is a recent graduate of the College of William and Mary and has served as the assistant editor of Gifted and Talented International for the past 3 years. E-mail: [e-mail]

Annie Xuemei Feng is the Research and Evaluation Director of the Center for Gifted Education at the College of William and Mary. Her research interests include curriculum effectiveness studies, program evaluation research, gender-related studies, and cross-cultural research. E-mail: [e-mail]

Table 1
Categories and Items within Each Category

Category            # of    Behavioral items
                    Items

Curriculum            5     set high expectations for student
planning and                  performance
delivery (CPD)              incorporated activities for students to
                              apply new knowledge
                            engaged students in planning, monitoring,
                              or assessing their learning
                            encouraged students to express their
                              thoughts
                            had students reflect on what they had
                              learned

Accommodations              provided opportunities for independent or
for individual        4       group learning
differences (AID)           accommodated individual or subgroup
                              differences
                            encouraged multiple interpretations of
                              events and situations
                            allowed students to discover key ideas
                              individually

Problem solving       3     employed brainstorming techniques
(PS)                        engaged students in problem
                              identification and definition
                            engaged students in solution-finding
                              activities and comprehensive solution
                              articulation

Critical-thinking     4     encouraged students to judge or evaluate
strategies (CRI)              situations, problems, or issues
                            engaged students in comparing and
                              contrasting ideas
                            provided opportunities for students to
                              generalize from concrete information to
                              the abstract
                            encouraged student synthesis or summary
                              of information within or across
                              disciplines

Creative-thinking     4     solicited many diverse thoughts about
strategies (CRE)              issues or ideas
                            engaged students in the exploration of
                              diverse points of view to reframe ideas
                            encouraged students to demonstrate
                              open-mindedness and tolerance of
                              imaginative, sometimes playful
                              solutions to problems
                            provided opportunities for students to
                              develop and elaborate on their ideas

Research              5     required students to gather evidence from
  strategies (RS)             multiple sources through research-based
                              techniques
                            provided opportunities for students to
                              analyze data and represent it in
                              appropriate charts, graphs, or tables
                            asked questions to assist students in
                              making inferences from data and drawing
                              conclusions
                            encouraged students to determine
                              implications and consequences of
                              findings
                            provided time for students to communicate
                              research study findings to relevant
                              audiences in a formal
                              report/presentation

Table 2
Specifications of COS-R Categories and Items Linked to Best Practice
Literature

Category                           Relevant studies of best practice

Curriculum planning and delivery   Good & Brophy, 1997; Guskey, 2000;
(CPD)                              NRC, 2000; Wang, Haertel, &
                                   Walberg, 1993.

Accommodation for individual       J. A. Kulik & C. -L. C. Kulik,
differences (AID)                  1992; Rogers, 2002; Tomlinson,
                                   1999; VanTassel-Baska, 2000;
                                   VanTassel-Baska, Bass, et al.,
                                   1998; VanTassel-Baska, Quek, et
                                   al., 2005.

Problem-solving strategies (PS)    Dods, 1997; S. A. Gallagher,
                                   Stepien, & Rosenthal, 1992.

Critical-thinking strategies       NRC, 2000; Paul, 1992; Paul &
(CRI)                              Elder, 2001; VanTassel-Baska, 2000;
                                   VanTassel-Baska, Quek, et al.,
                                   2005; Wenglinsky, 2000.

Creative-thinking strategies       Cramond, 2001; Davis, 2001;
(CRE)                              Treffinger & Isaksen, 2005.

Research strategies (RS)           NRC, 2000; Reis & Renzulli, 1992;
                                   VanTassel-Baska & Little, 2003;
                                   Wenglinsky, 2000.

Table 3
Observable Evidence for Each of the Subject Areas
Might Be as Follows:

Math             Asked boundary/condition questions about proof/
                 theorem such as "Under what conditions will this
                 proof hold up and under what conditions will it not?"

Science          Asked questions such as: "Were the results
                 replicable?"; "Were the data reliable?"; "Was the
                 experiment well-designed?"

Literature       Encouraged students to form interpretative hypotheses
                 and test them on further reading or subsequent
                 readings by applying criteria of plausibility and
                 consistency.

Social Studies   Asked questions about the implications of context for
                 understanding a primary source document.

Foreign/Second   Asked questions about an author's purpose and
Language         assumptions.

Table 4
The COS-R Reliability

                              Internal consistency Alpha

                      Pilot study   1st observation   2nd observation
COS-R category         (N = 50)        (N = 72)         (N = 58-62)

Curriculum planning       .54             .79               .67
  & delivery

Accommodations for        .69             .68               .73
  individual
  differences

Problem solving           .73             .82               .94

Critical-thinking         .66            65.00              .78
  strategies

Creative-thinking         .63             .86               .77
  strategies

Research strategies       .89             .83               .83

Overall                   .92             .91               .93

Table 5
COS-R Content Validity

Dimension                            Intraclass
                                     coefficient

Importance of instructional
behavioral items in the
Teacher Observation scale               .86

Clarity of language in
describing items in the
Teacher Observation scale               .99

COS-R content validity                  .98