Using data to inform instruction: while test scores are accepted as a measure of school effectiveness, the data should also be used in a more student-centric way to inform targeted instruction.

Welcome to California's Accountability Cafe! Allow me to recommend our soup du jour, Acronym Bisque. The base is complex--tangy with somewhat of a bite, consisting of a blend of AYP, AMOs, API, CAT/6, CSTs, CAHSEE, CAPA and a pinch of CELDT.

With the implementation of the Federal No Child Left Behind legislation, we were all introduced to a variety of acronyms designed to simplify the discourse surrounding the mandate. Similarly, as we continue to use the Academic Performance Index as a state measure of accountability, the opportunity to use shortspeak (as opposed to shorthand) has been afforded us through the coining of terms such as CAHSEE, CAPA and CELDT, to name a few.

These acronyms, like the legislation they represent, are presumably intended to simplify and make more "user-friendly" the complex task of measuring student learning and identifying schools and districts demonstrating annual growth, or what is more commonly referred to as "adequate yearly progress."

Or not.

It should come as no surprise that the ingredients of California's Acronym Bisque, by and large, represent the tests used to evaluate schools and districts. With very few exceptions, the indicators used to identify underperforming schools take the form of multiple-choice test results and student writing samples.

While student test scores are generally accepted as an appropriate proxy for measuring the effectiveness of schools and districts, it has been argued by many that these data should be used in a more student-centric fashion to inform and promote targeted instruction sensitive to the needs of children.

It is this perspective that challenges my colleagues and me to go beyond the carefully crafted acronyms, the mellifluous discourse and the intellectual circumlocution often associated with sweeping legislation and focus on that which, ironically, appears to have received about half the attention afforded the legislation itself--our students.

Having embraced this challenge, the San Diego County Office of Education's Assessment, Accountability and Research Unit provides professional development opportunities that are student-centric with respect to classroom assessment practices, data analysis strategies and the use of test scores to inform instruction targeting individual student needs.

Teacher preparation

Maintaining our focus at the student level, our initial efforts included a workshop designed to provide teachers and administrators with the necessary tools for identifying precisely what their students were learning and what gaps existed.

After working with teachers at all grade levels and content areas, we've come to understand that the majority of teachers have had little or no formal training in the area of classroom assessment and evaluation. In fact, after more than 50 workshops representing more than 2,500 participants, we have yet to have more than 10 percent of an audience indicate they had received a semester-long course on classroom assessment and/or measurement.

Given the high-stakes nature of assessment and accountability in public education today, this critical need in teacher preparation has helped to define our work for the foreseeable future. Our workshop, "Developing Standards-Based Assessments for the Classroom," addresses the various types of tests commonly used in classrooms as well as their respective components.

Participants are introduced to assessment concepts and vocabulary such as content validity, reliability, test-item development, item selection and the use of test results to inform instruction. In a nutshell, this workshop is designed to touch on the major aspects of developing and using standards-based classroom assessments to inform instructional effectiveness, student mastery and remediation efforts.

The framework underlying our work in this area is best reflected in the relationship among curriculum, instruction and assessment. It is the relationship among these three domains--each reflecting the state standards--that provides the foundation for developing targeted classroom tests.

It is our belief that aligning curriculum, instructional practice and assessment strategies is a critical step toward accurate identification of learning and instructional deficits, identifying promising practices and promoting student learning.

I've often stated that teachers should never find themselves in a situation where they're teaching to a test. However, all teachers, in order to evaluate their instructional effectiveness and students' learning, should "test to the teach."

Testing to the teach

This fragment of thought helps characterize the alignment of assessment, curriculum and instruction. Embedded also are a number of important considerations that must be entertained when developing sound, standards-based tests.

First, a clear understanding of the curriculum and instructional resources used to exercise the standards defining the lesson must be demonstrated.

Second, recognition of instructional intensity, or time spent teaching each standard or set of standards, is essential for establishing content validity and appropriate weighting of test items.

And third, the selection and/or development of appropriate test items must accurately reflect the content and skills articulated in the standards being evaluated.

The first two considerations are captured in the development of a test blueprint or table of specifications. Workshop participants are taught how to create a table of specifications reflecting their most recent teaching. This authentic exercise gives teachers a tool that can be immediately implemented in their classrooms upon their return.

We're continually amazed at how much more empowered and confident teachers feel about their testing practices after completing this part of the workshop. For many, it is the first time they've received any formal instruction on how to systematically evaluate their students and themselves in a way that truly reflects their teaching.

Assuming success in addressing these considerations, the more challenging and time-consuming work of writing good test items and selecting the most appropriate item format awaits.

Using Bloom's "Taxonomy of Educational Objectives" (1956) as a backdrop, teachers are instructed how to identify the level of learning being articulated in the standards and which item types are most appropriate for measuring each level (Linn & Gronlund, 2001).

Bloom's framework outlines six levels of learning, starting with knowledge, followed by understanding, application, analysis, synthesis and lastly, evaluation. After determining which level of the taxonomy the teacher is addressing, strategies for selecting the most appropriate item type are provided.

As an example, true-false and matching exercises are item types used at the knowledge level, as they typically require mere recall of names, dates or other details to generate a correct response. This is in contrast to using an essay-type open-ended question measuring at the evaluation or synthesis levels, which require higher order reasoning, writing and other complex skills. Helping teachers understand when and how to use different item types enables them to self-reflect and think critically about their testing practices.

Classroom assessment typically takes the form of objective tests (multiple choice, fill-in-the-blank, matching exercises). However, some learning outcomes require the use of portfolios, authentic assessments, observation or performance tasks. Knowing when and how to use these subjective or qualitative evaluation strategies speaks again to the importance of aligning curriculum, instruction and assessment. These strategies, coupled with the more common paper-and-pencil tests outlined above, can provide a wealth of information to the classroom teacher.

Going beyond the classroom

Taking one step back from the classroom but maintaining a student-centric focus, we at the SDCOE have developed a data analysis tool that breaks down the State Testing and Reporting (STAR) program data, reconfiguring and presenting the data in a very user-friendly fashion. The tool is called A2Q: Answers to questions using student achievement data.

One of the greatest challenges we've uncovered regarding the huge amount of data schools receive as a result of state testing is how to use it. In fact, many administrators and teachers would love to simply be able to access it off the data disk sent to them courtesy of the CDE. It was this cry for help that catalyzed the development of A2Q.

To spare readers the details behind the programming that reconfigures the STAR data file, suffice it to say that upon receipt of a data disk, A2Q can produce a Microsoft Excel file containing student-level data capable of being manipulated using Pivot Table technology in less than 30 minutes. In fact, in most cases, districts of 20,000--30,000 students can be processed in less than 10 minutes.

So once one has access to the data ... now what? We've always believed that the collection and dissemination of data often amounts to nothing more than a lot of activity. We needed our efforts to be more than that--to be actionable. In other words, we wanted to provide a tool that would inform teachers and administrators as to the specific needs of individual students. In addition, our intent was to provide information on a less granular level that looks at program and instructional effectiveness among groups of students.

The user interface for A2Q is simply an Excel spreadsheet. Within the Excel application is a Pivot Table option that allows users to select various data elements and examine their relationships. Pivot Tables are generally used to disaggregate and examine data so that patterns and trends among variables can be more easily identified.

As an example, through a series of clicks on a mouse one can create a Pivot Table that identifies how many fourth-graders attending a particular school who are non-native speakers of English scored far below basic on the English/language arts CST. Moreover, by clicking on the cell entry within the table, all of the individual student data associated with those meeting these criteria will be presented and displayed.

This is a very powerful tool that provides student-level data identifying performance levels across all strands within each of the math and English/language arts CSTs. For example, a student who scores proficient overall on the math CST may still need remediation in one or more of the content strands contributing to the overall math score. A2Q provides strand-level information that identifies these gaps so that targeted support can be provided to individual students.

Using data to Inform practice

It is our goal to provide teachers and administrators with tools that support their efforts at using data to inform their practice. With all the energy currently being spent on defining how schools and districts are to be measured for accountability purposes and how they're to manage the sanctions associated with demonstrated underperformance, not to mention the logistics inherent in the testing process itself, one wonders what intellectual resources remain for discussing and implementing strategies that may prove fruitful to students.

We've tried to describe examples of how teachers and administrators can better acquire and use student test scores to improve the learning experiences for all kids. We're hopeful that these strategies, among countless others, will help to bring about the fundamental change needed to ensure that all students meet the academic demands placed before them.

Leaving no child behind requires that care, attention and consideration be afforded all children. Success at becoming a life-long learner is something that can be achieved by each of us. As our focus shifts from meeting the letter of the legislation to meeting the needs of our kids, greater success among all students will be realized.

References

Linn, R.L. & Gronlund, N.E. (2001). Measurement and Assessment in Teaching, 8th edition. Englewood Cliffs, NJ: Prentice-Hall.

Bloom, B.S., et al (Eds.). (1956). Taxonomy of Educational Objectives: Handbook I, Cognitive Domain. New York: D. McKay.

Dennis A. Johnston is the director of assessment, accountability and research for the San Diego County Office of Education. John T. Lawrence is special projects coordinator for the San Diego County Office of Education.