Data analysis demystified: standardized tests don't do what matters most -- meeting the immediate needs of individual students. That's why schools must design, deliver and analyze additional sources of meaningful student data.

When we design curriculum, choose instructional strategies and prepare for program implementation, we should also plan for data analysis. There are two essential perspectives for data analysis: summative analysis and formative analysis. Summative analysis is collecting and presenting information that is necessary to make final statements and judgments about the value of an activity, usually at the end of the activity's implementation. Formative analysis is the continuous monitoring of short-term results and procedures to provide ongoing information to improve student achievement.

In other words, summative analysis is using student achievement results to determine how well something worked after an activity is completed. Formative analysis is using today's student achievement results to help us determine what we will do tomorrow.

With state-mandated tests, our vision is often skewed because we only see the summative aspect of assessment and data analysis. To counteract this situation, teachers, schools and school districts must actively collect and analyze other formative student achievement data to help them improve instruction and student learning.

Comprehensive data analysis efforts should certainly include scores from all the state testing programs; however due to the nature of the tests, state test scores do not reveal enough detail about individual students' mastery of the specific content of the assessment.

Calkins, Montgomery & Santman (1998) stated that "standardized tests don't measure what matters most." State test scores do not reveal how the test content relates to student mastery of specific state standards. Therefore, their use as formative tools is limited.

The nature of the state tests may actually impede our ability to make formative decisions about instruction. State tests are not administered frequently enough to help school sites drive program implementation decisions or determine program effectiveness. State test scores do not assess enough of the standards to allow districts to conduct a comprehensive summative analysis of their local curriculum. Therefore, local school districts should work with teachers and site administrators to design, deliver and analyze additional sources of meaningful student performance data that are aligned to state standards.

According to Douglas Reeves (1998), "Standards without standards-based assessments are merely a very expensive and time consuming pep talk."

It is a mistake to think of data simply in terms of test scores. According to Webster's Unabridged Dictionary, data are premises upon which things can be argued or inferred. In education, these premises are meaningful bits of information upon which educators can base their decisions about the effectiveness of instructional strategies, the relevance of the curriculum content, and the subsequent impact on student achievement. While this may sound simple, producing bits of meaningful information regarding student performance is a major challenge for most school districts.

Assessment literacy

The first step in using data to improve student learning is to help school staff members become proficient at assessment and data analysis. Michael Fullan (1991) said, "The crux of the matter is getting the right people together with the right information." Even if districts successfully produce meaningful formative student achievement data, placing the data into coherent formats that allow for simple analysis is still a major challenge.

Once local assessments are developed, teachers need time and training. Districts must train their teachers to properly analyze multiple local assessment results and apply their findings to guide their instruction. Since constructive data analysis incorporates more than merely looking at test scores, teachers need training on different methods to assess their students' progress toward mastering standards.

According to Richard Stiggins (2001), "Assessment Literacy is the key." Teachers must learn to look at student work and plan their lesson based on what they see. Teachers must learn to use assessment tools such as standards-based report cards and scoring rubrics. They must be able to analyze detailed local reports that indicate their students' responses to specific standards-based assessment items.

A well-designed district math assessment, for example, that is closely aligned to classroom instruction, the district's curriculum and the state standards is useless if teachers do not have the training to analyze the results, do not have the time to act upon the information revealed in the analysis, and do not have reporting methods that allow them to reliably describe each student's performance.

All meaningful data analysis begins with good assessments at the classroom level. All assessment should be designed to help teachers accurately measure their students' academic standing relative to the standards, the local curriculum and their own instruction. Douglas Reeves (1998) calls this congruence.

The data should be formatted in ways that the teacher and the student can understand. All staff members must act upon this information to help form their instructional decisions. Then the school can use the data as valid indicators of student progress. The school can aggregate the data into reports that determine program effectiveness, indicate instructional efficiency, and document public accountability.

Again, this may sound simple; however, problems such as collecting and collating the data, turning results into action plans and generating clear and valid reports still remain a sizable challenge.

It is essential that the district office maintain an active role in organizing the curriculum, developing the assessments and reporting the results. The effort required to develop assessment and data analysis systems is daunting; the district office can organize resources to facilitate the development of these standards-based tools.

It is important for all students to receive a similar instructional program with consistent content. The district office is the unit that helps develop the tools to provide consistency across their schools and consistency within the classrooms. Frequent local assessments that are processed at the district level with coherent, useful reports back to teachers and school sites allow each school to use current data to assess student progress and analyze program effectiveness during implementation. This type of formative data analysis is a fundamental element of any school improvement effort.

Once a district develops a consistent curriculum as well as valid and reliable assessment instruments, how will it manage the data? Depending on the scale of the operation and the sophistication of the organization, there are many methods. Technology enables districts to provide teachers with current student performance data.

Teachers should receive detailed, item-by-item local assessment results for each student, as well as whole-class reports. Teachers should also receive reports that align student responses to each test item to state standards. If the assessments are aligned to state standards and they are sequenced to the units of instruction outlined in the district's curriculum guides, the detailed reports (data) become powerful tools for the teachers and the site administrators. Teachers can use the data to form their subsequent instruction and identify topics for remediation and enrichment. With careful flagging, student progress can also be linked to participation in specific programs or interventions.

This enables the district and the sites to use the local assessment data to evaluate program effectiveness. For example, several sites within a local school district implemented a program designed to improve reading through strengthening auditory processing speed. The program was research-based and was proven to be effective, but appeared to be relatively expensive. The pressure to demonstrate results was tremendous, and evaluation could not wait until the end of the year's state test results.

Following the first three months of implementation, the district analyzed local assessment results for reading comprehension, writing (content and conventions) and mathematics. Although the intervention was designed to improve reading comprehension, the data indicated that math scores and writing conventions also improved at a faster rate than for similar students who did not participate in the program. These local data allowed the district to demonstrate the program's effectiveness and expand its implementation prior to the end of the academic year.

A second example of formative data analysis resulted from the evaluation of a local district program designed to supply each school site with trained literacy experts. Careful examination of the data revealed that the literacy experts were most effective when they worked within a specific delivery model at specific grade levels with students who had specific attendance patterns. The program was then able to focus resources and identify strategies that better served all the identified students. The assessment data helped the sites determine the greatest intervention impact early in the program and allowed them to adjust their implementation for maximum effect.

Neither formative nor summative data analysis can enhance instructional quality if one type of analysis is used in isolation of the other. Summative analysis is essential to determine program effectiveness. Formative evaluation is essential to ensure correct implementation and to identify necessary adjustments during implementation. With either purpose, it becomes imperative that we use data to help us improve our educational efforts and our ultimate service to students.

If we rely entirely on an analysis of data that merely reflect a summative measure of our schools' effectiveness, we are likely to ignore the immediate needs of individual students. The adjustments we make to improve our schools must be based on a comprehensive analysis of all available data, the continuous pursuit of more information, and the subsequent refinement of our analytical skills.

STATEMENT OF OWNERSHIP, MANAGEMENT AND CIRCULATION
(REQUIRED BY 39 U.S.C. 3685)

1.    Publication Title:            Leadership

2.    Publication No.               0282740

3.    Filing Date                   October 2001

4.    Issue Frequency               Sept./Oct., Nov./Dec.,
                                    Jan./Feb. Mar./Apr.,
                                    May/June

5.    No. of Issues Published       5
      Annually

6.    Annual Subscription Price     $50.00

7.    Location of known office of   1517 L Street,
      publication:                  Sacramento, CA 95814
                                    Sacramento County

8.    General Business Office of
      Publisher:                    1575 Bayshore Highway,
                                    Burlingame, CA 94010

9.    Publisher:                    Association of
                                    California School
                                    Administrators 1517 L
                                    Street, Sacramento, CA
                                    95814

      Editor:                       Susan Davis
                                    1517 L Street,
                                    Sacramento, CA 95814

      Managing Editor:              Susan Davis
                                    1517 L Street,
                                    Sacramento, CA 95814

10.   Owner:                        Association of
                                    California School
                                    Administrators
                                    1575 Bayshore Highway,
                                    Burlingame, CA 94010

11.   Known Bondholders,
      Mortgagees, and Other
      Security Holders Owning
      or Holding 1 percent of
      more of total amount of
      bonds, mortgages or other
      securities:                   None

12.   For completion by nonprofit
      organizations authorized      Has not changed during
      to mail at special rates:     preceding 12 months

13.   Publication Name:             Leadership

14.   Issue date for circulation    May/June 2001
      data below:
15.   Extent and nature of          Average no.    Actual no. copies
      circulation                   copies each    of single issue
                                    issue during   published
                                    preceding      nearest to
                                    12 months      filing date

a.    Total number of copies        17,763         18,400

b.    Pad and/or Requested
      Circulation

      (1) Paid/Requested            17,016         17,659
      Outside-County
      Mail Subscriptions

      (2) Paid In-County                 0              0
      Subscriptions

      (3) Sales Through
      Dealers and Carriers,
      Street Vendors
      Counter Sales, and
      other Non-USPS Paid
      Distribution                       0              0

      (4) Other Classes Mailed
      Through the USPS                   0              0

c.    Total paid and/or             17,016         17,659
      requested circulation

d.    Free distribution by mail

      (1) Outside County                 0              0

      (2) In County                      0              0

      (3) Other Classes Mailed
      Through the USPS                 160            160

e.    Free distribution outside        150            150
      the mail

f.    Total free distribution          310            310

g.    Total Distribution            17,326         17,969

i.                  Copies not distributed     437        431

                    Total 17,763    18,400

      Percent paid and/or
      requested circulation         98%               98%

16.   Statement of Ownership will
      be printed in the
      Nov./Dec., 2001 issue of
      this publication.

I certify that all information furnished on this form is true and
complete.

Susan Davis
Editor

References

Calkins, L., Montgomery, K., & Santman, D. (1998). A Teacher's Guide to Standardized Reading Tests. Portsmouth, NH: Heinemann.

Fullan, M.G., with Stiegelbauer (1991). The New Meaning of Educational Change. New York: Teachers College Press.

Reeves, D. (1998). Making Standards Work. Denver: Center For Performance Assessment.

Stiggins, R. J. (2001). Student Involved Classroom Assessment. Columbus, Ohio: Merrill Prentice Hall.

Dave Ackley is director of administrative support and student programs for the Fontana Unified School District.