The power of technology: statistical tools for finding answers to student achievement questions.

A middle school teacher wrote to her intermediate service agency mentor, "We had everybody here today and my pre-algebra class broke a record today: 140 out of 155 possible. The previous record was 134. We are going to celebrate tomorrow. I just wanted to share my excitement with you. All of my classes have improved throughout the year. I wish I had known about this before!"

What the teacher was referencing was the process of quizzing students weekly on end-of-the-year standards. These non-graded quizzes randomly sample concepts to be learned from the total body of the year's content. With the help of technology, she was able to easily track student progress throughout the year, identifying areas in which students needed additional help and support. The technology helps educators manage and lead learning with precision.

This emphasis on setting high academic standards and measuring students' progress in meeting those standards dictates that educators afford assessment data a greater role in decision making than ever before. Comprehensive student achievement data are vital to determining how students are progressing toward mastering standards and meeting adequate yearly progress requirements.

Consider this: If a state has an average of 50 academic standards each for language arts, math, science and social studies per grade in a school district with 13 grades and 5,000 students, district leaders must assess student performance on 13 million data points. That's an overwhelming responsibility.

With the proper framework in place, however, today's technology can make possible this daunting task of both leading and managing students and standards. This is true for the teacher with more than 100 students and the administrator responsible for 5,000 students.

Necessary Framework

Before technology can be used to its fullest potential, the school district must have in place a solid base of non-technology-related structures, such as clear standards and expectations, a comprehensive assessment plan and professional development.

* Standards alignment and expectations: Students should understand what they need to know and demonstrate in each subject in each grade level. That means standards and expectations should be written in easy-to-understand terms. Teachers must understand and support high expectations for students and accept their responsibility to promote their success. That responsibility includes reinforcing past learning, teaching current content standards, and preparing students to master the material at the next level.

* Comprehensive assessment: Districts need a formative weekly or biweekly assessment process that helps educators easily identify students' achievement levels and map their progress toward mastering standards throughout the year.

One strategy suggests that teachers assess students each week on randomly selected concepts from the curriculum. Approximately 70 percent of the questions come from the current year and 30 percent from previous years.

For example, suppose 10th grade geometry students are expected to master 75 new concepts during one year. The weekly quiz could include seven items randomly selected from the total standards for the current year, one item from K8 geometry, one item from Algebra I and one item from middle school general math. A crucial aspect of a comprehensive assessment program is removing "permission to forget" content from prior courses.

Students should be expected to have 30 percent of the items correct the first week of school, as that is the percentage of items students learned in past years. Student growth then is measured from the expected 30 percent correct the first week of school to the expected 100 percent correct by year's end. These assessments are not graded, but they are graphed for individual students and for classes and schools as a whole.

Rochester, Ind., under the leadership of recently retired Superintendent Bob Poffenbarger, became the first district in the nation to structure these quizzes using the 70/30 formula. The district's 6th-grade students answer seven questions each week randomly sampled from the complete year's content, two questions from grade 5 curriculum and one from grade 4. Jenks, Okla., has followed this same process for both language arts and mathematics.

Harnessing Technology

Teachers and administrators must understand and implement the process of improving student achievement so data show movement from the L-shaped graphs through the bell to the J-shaped graphs, know how to use data efficiently and effectively and know which graphs are most meaningful. A two-day seminar, combined with scheduled follow-up, is usually sufficient.

Staff development with regard to the technology itself is two-fold: (1) helping district leaders define consistent parameters for data in every grade and every subject and (2) helping teachers and administrators use specific software. One strategy is to have highly trained individuals in each school plus one centrally located expert who can serve the needs of the entire district.

With these structures in place, school leaders can use the power of technology to answer questions that will help them guide their students toward achievement. The technology required to facilitate a basic assessment program like the one described is minimal.

Clearly all teachers need access to a high-speed Internet connection, a computer powerful enough to handle a variety of software programs, and a printer--preferably in their classroom--so they can print the graphs that show students' learning progress. The recommended software includes a spreadsheet program such as Excel. The advantage of Excel is most districts already have a license. The disadvantage is that somebody must program it to meet district needs as aggregating scores from classroom to classroom and school to school can prove to be a daunting task. Other data management software packages are available to track and report student learning progress and to aggregate and disaggregate data according to NCLB categories.

Definitive Answers

With a solid assessment process and the requisite technology in place, school leaders will be able to find a definitive answer to questions they could only guess about before.

* Are our expectations high enough?

This question is answered with a simple column graph using data from a pre-test or, as suggested above, the data from the first week's 70 percent new content assessment.

On a spreadsheet, record or import the number of students with 0-10 percent correct, the number with 11-20 percent correct, the number with 21-30 percent correct and up to 91-100 percent correct. Create the column chart and study the shape of the graph. If the expectations are high enough, the curve will be to the left on the chart--an L-shaped curve. In other words, the majority of the students have not mastered the content at the beginning of the school year. If the results show a bell curve, then the expectations are too low, as many students have already learned the material and may be left unchallenged after the first few weeks of school. If the graph is a bell, teachers must immediately raise year-end expectations.

* In general, did the students begin the year knowing the content from prior years?

If the pre-test during the first week of school includes content from prior years or if the 30 percent prior-year review assessment strategy is implemented, graph the student results for the review portion separately.

The graphed results for review items should be in the shape of a J, indicating the students remembered the content following summer vacation. In this case, teachers may decide where new content begins in the textbook, start instruction with that content and avoid spending valuable time on unnecessary review.

If the graph shows a bell curve or an L shape, the students probably didn't retain the content learned in the prior years. Instead, they learned the content for the chapter tests only. The students accepted the unspoken "permission to forget" as the norm. It's time for review.

* Which students did not master the prior year's content? Which students have already met end-of-year standards?

A simple sort function in a spreadsheet program probably provides the best example of how technology can help educators understand student achievement in their district. The program can quickly tell administrators which students are beginning the school year without adequate mastery of the prior years' content and which have met end-of-year expectations. The key is to continuously compare what students know today with what they are expected to know and be able to do by the end of the year.

For example, 1st-grade teachers who want students to be able to read all 220 Dolch sight words by the end of the year might randomly select 12-15 words a week for the students to read. If a student reads all randomly selected words for seven consecutive weeks, the teachers can be confident that the student knows all 220 words and, after seven weeks, will change the assignment to challenge the students for the remainder of the year.

In Jenks, Okla., principals and district officials view the sorted data displayed on a scatter diagram every week. The large spreadsheet has a dot for every student in the school or school district for every week. A mere click of the mouse provides student names for easy identification of who is where in the learning process. The data reflect student knowledge today, not last month or last year, so education leaders have a real-time view of student achievement.

* Are we on target to meet standards?

Measuring real-time student learning against end-of-the-year expectations is the key to knowing whether students will meet standards. For example, the middle school teacher quoted earlier knows her class has met 90 percent of the standards with 94 percent of the time used.

* What problems need to be addressed?

Technology helps educators determine which students may be behind as well as which areas of the curriculum are causing students to struggle. Aggregation of data provides the big picture so necessary for leading learning. Disaggregation helps educators manage the specifics.

For example, disaggregation of classroom data by gender showed teachers at Rochester Middle School in Rochester, Ind., that the boys and girls were performing equally well in mathematics in grades 6 and 7 and that the boys were outperforming the girls in grade 8. Further disaggregation by economic status showed that the girls who lived in poverty were causing the gender difference. The girls from low socioeconomic environments were 3-5 weeks behind the non-poverty boys and girls. With this information, educators are able to manage resources to target those students who need extra support.

* Are students struggling because some concepts are more difficult for them to learn than others?

While many educators believe that item analysis is helpful to determine how well students are learning specific concepts, few practice it because it is time-consuming. Technology, combined with sampling, is the answer.

Using www.randomizer.org, teachers can randomly select five student papers from each period or class and itemize the errors to create an item analysis for these papers. Random selection of five students, plus item analysis and teaching to the errors, will produce better results than 100 percent inspection with no item analysis.

Once practice is over and grades need to be assigned, teachers will, of course, read 100 percent of the final papers. When administrators need to communicate item-analysis results for large sets of data, they should consider the Pareto Chart. It is easily made with QI Macros, an add-on to Excel. To make it work the administrator needs the raw number of errors per strand or error type.

Final Note

In the post-NCLB world, technology is available to answer the essential questions about learning and achievement. Are expectations high enough? Will we meet end-of-the-year standards? Did we end the year with a high success rate (a J curve)? Which students and which concepts are causing problems?

The foundation for unleashing the power of technology, however, lies with the decisions made before technology is added to the mix: standards alignment, staff development and assessment.

Resources

Lee Jenkins recommends the following resources related to his article:

* Improving Student Learning by Lee Jenkins, Quality Press.

* Permission to Forget by Lee Jenkins, Quality Press.

* QI Macros for Excel, statistical processing software manufactured by KnowWare International.

* Free demo DVD available from Jenkins that may be used for staff development.

Lee Jenkins is president of From LtoJ Consulting Group, 11445 E. Via Lindo, Suite 2, PMB 481, Scottsdde, AZ 8559. E-mail: [e-mail]