Feedback at 360 Degrees.

Avoid a do-it-quick approach if you want performance data that are valid and reliable

On school boards these days, one or more members probably have experienced the power of 360-degree feedback in their own careers. Many school administrators, in their preparation programs or in professional reading, have learned how multi-source personnel evaluations are much better than a unilateral rating by the boss.

Public and independent schools have used forms of multi-rater feedback for evaluating job performance long before Teams Inc., a consulting group in Miami, trademarked the term "360-degree feedback" in 1973. Most often, it was small-scale, personalized and occasional.

A teacher might survey her students with a questionnaire modified from a professor evaluation form completed in graduate school. Or a principal might ask teachers, "How am I doing?" and "How can I improve ?" Superintendents might solicit feedback from a parent advisory group.

Hasty Shortcuts 

Unfortunately, many school districts that have moved into the use of 360-degree feedback have taken shortcuts, skipping the careful steps that provide validity and reliability to the data collected. The steps that my colleagues and I at the School Improvement Model Center at Iowa State University consider most essential are these:

* Using a collaborative design team that involves representatives of all stakeholder groups;

* Starting with feedback to the top (the school board, superintendent, members of the superintendent's cabinet, principals);

* Running a small pilot test to debug the procedures before applying the process widely; and

* Avoiding early publicity that only heightens the fear of feedback for teachers and alarms parents about retribution for critical comments.

It takes time to gain an appreciation of the objectivity of 360-degree feedback. Teachers fear parents; parents fear teachers. Many principals have no idea what their faculty will say if given the chance. The new push for greater accountability encourages haste. Board members and state legislative leaders (as well as public education's critics) want data now.

The let's-do-it-quick approach results in mass mailings, paranoia and scant returns that tend to be slanted to the negative side by angry respondents. Haste also causes the spotlight to focus on feedback to teachers from students and parents, the most difficult part of 360-degree feedback for schools.

Essential to the Task

The process of 360-degree feedback is a sampling technique. It is not a 100 percent survey. Furthermore, 360-degree feedback can be used at three levels: (1) for developmental purposes (for the employee's eyes only), (2) for appraisal and (3) for compensation.

The prudent school districts stay at level one for at least one year, sometimes two. Only in rare instances will the third level be applied. Cave Creek, Ariz., Unified School District uses student feedback in determining career ladder promotions, while the West Des Moines, Iowa, Community Schools uses 360-degree feedback to determine group incentive pay for administrators.

The School Improvement Model Center has assisted four superintendents in establishing 360-degree feedback to administrators and teachers in the past two years. While districts each have some unique aspects to their programs, they also have strong common elements.

Robert Holmes, superintendent of Riverhead, N.Y., Central Schools, and Dennis Pope, superintendent in Bedford, N.H., used a top-first approach to establish feedback to the board, superintendent and principals to set a good example.

In both instances, a random sample of 25 percent of all parents and guardians was drawn from the enrollment file and a direct-mail survey was conducted. A second mailing was needed. This time, 10 percent were randomly drawn. Combining the returns from both mailings obtained a 25 percent response rate--with a confidence level of 4 percent.

Additional questionnaires were made available through meetings at the school sites. These were tabulated separately and proved to be more negative than those produced by sampling techniques. A stakeholders' committee in each district planned the process, and our center served as the data processing and analyzing agency.

Donna Jemilo, superintendent of the Burlington, Vt., Public Schools, and Marilyn Semones, superintendent of the Camp Verde, Ariz., Unified Schools, also sought 360-degree feedback for all employees, but they linked feedback from parents to student feedback to teachers.

Worth Emulating

Each of the four superintendents became a connoisseur of team feedback. They learned that 360-degree feedback can enhance a district's performance evaluation system by helping gain agreement on expectations, using a broader range of information and facilitating open discussion.

From their experiences, we have summarized the following procedures for effective implementation:

* Seek an array of respondents. For example, insights about a principal would be sought from teachers, students, parents, peer principals, support staff and central administration. For feedback about a teacher, students, parents, peer teachers, support staff and building administrators would be asked.

* Create a sampling procedure that fits the need. In private industry, 360-degree evaluation uses as few as four respondents in each group, but educators ought not be so limiting. Principals typically try to use about 10 respondents. However, as several of our pilot tests revealed, all teachers want the chance to give feedback to the principal. When evaluating teachers, use all of an elementary teacher's students and two sections for each secondary school teacher. Survey instruments can be tailored for grades K-2, 3-5, 6-8 and 9-12.

* Put the process in place over three years. Evaluate administrators first, then teachers, using all respondents except parents. In the third year, parents should be included.

* Publicize good examples. The first 360-degree feedback to Semones, superintendent in Camp Verde, Ariz., which was quite positive, was published in the local newspaper. Teachers and administrators could see that the information was important and taken seriously by the superintendent and board. In West Des Moines, principals shared their aggregated feedback with their faculties. The district had used 360-degree feedback for administrators for three years before introducing the process as a part of teacher performance evaluation.

* Apply sampling theory for parent surveys. A random sample that generates a 25 percent response yields a confidence level of plus or minus 4 percent. When asked for their feedback, parents should be informed that their students already have participated to encourage their part.

* Use an outside authority for validity. The School Improvement Model Center provides analyses and reports to school districts, including validity checks, reliability measures and item discrimination.

* Use the feedback for improvement. Some school districts provided coaching by external consultants, sometimes in groups, sometimes one-on-one. For example, during the spring of 2000, I met with all of the principals individually at Waterloo, Iowa, after processing their 360-degree feedback. Using a scripted process, we examined strengths, pinpointed shortcomings and wrote a growth goal. The script and growth goal forms were provided in multiple copies. Later, many of these principals used the same procedures to coach their direct reports.

The Nonresponse Issue

In our pilot tests, some parents, students and other respondents returned a blank instrument or incomplete data. This raises the question of how to treat nonrespondents. J.D. Power and Associates monitors the quality of automobiles sold in America (16 million or more per year) based on 90,000 surveys. Their conclusion about nonrespondents will serve school districts too.

Because of random sampling, all in the sample have an opportunity to reply, according to J.D. Power and Associates. "Those who did not are neither more dissatisfied nor more satisfied than the respondents."

Richard Manatt is director of the School Improvement Model Center, Iowa State University, Ames, Iowa 50011.