The Tenuous Nature of Superintendent Evaluation.

Reluctant to use student outcomes, boards try more detailed assessments of system leaders' performance

The first time Pat Ruane was evaluated as a school superintendent it did not come out quite the way she and her school committee had hoped.

At the time, Ruane was running the Needham, Mass., school district. Members of the school committee (what Massachusetts calls school boards) marked her on a scale of 1 to 5 on an assortment of measures.

This is the way many school boards evaluate superintendents, according to recent surveys and interviews with administrators across the country. The process is often very simple. Forms are filled out by each school board member and sometimes summarized by the board chair. The board then discusses the results with the superintendent once a year during an executive session. Goals for the new year are agreed upon, and everybody goes home.

In Needham, Ruane had worked hard and made progress. The school committee gave her high marks. Its members conscientiously tried to reflect their opinions and those of the community, but the evaluation process was not something with which they had much experience or training, particularly in the way it would be perceived by the public.

Some members, wedded to a grading philosophy that schoolchildren know too well, did not want to mark Ruane so high that she did not have room to show improvement. So there were many 4's that might otherwise have been 5's. A local newspaper decided to run a story saying the district had, at best, a B-plus superintendent.

When they saw the story, Ruane says, the board members "really felt terrible. It was not their intent to give me a B."

Four years later, Ruane is midway through the second year of her new job as superintendent of the Lexington, Mass., schools. This time her board evaluation has been a very different process, a sign that changes are coming to this very old and often shortchanged part of school board routine.

The marks Ruane received in Lexington were also very high, but they came not in quick marks of 4s or 5s, but in a seven-page report that summarized an exhaustive review. The school committee members asked questions not only of themselves, but many other people involved in the community's schools.

"The school committee interviewed 23 people with whom the superintendent had direct contact," says Susan Elberger, whose term on the committee expired after the Ruane evaluation was done. "These included all her direct reports (principals, central-office administrators and others), presidents of all five unions in the school system, the town manager and a member of our board of selectmen, In addition, each of the five school committee members wrote an evaluation, as did the superintendent."

Critiques in Detail

Not many school boards are willing or able to put such effort into grading their superintendents. What the Lexington school committee did is often called a 360-degree evaluation, one that looks at the person being assessed from every possible angle. (A complete 360-degree assessment would have queried parents and students, which Lexington did not do.)

Many superintendents and personnel experts say school boards are taking the process more seriously than they have in the past and are moving toward longer and more detailed reports.

"Evaluations have more hard data, are less subjective and more objective, and the indices that the school boards are looking for are those measures that really make a difference and are clearly designed to do more for children," says Jim Warren, a superintendent for 16 years in two Illinois districts and now a partner in the Bickert Group, an executive search firm.

Edward Costa, superintendent in East Longmeadow, Mass., says one part of his evaluation assesses how he has done in reaching 47 objectives; another part measures his progress in meeting long-term strategic goals. "When used together they have proven very successful," he says.

But despite this trend, experts say, the majority of superintendents still find the evaluation process not much more detailed, time-consuming or helpful than a credit check at Sears. Nor do evaluations yet draw much notice outside the boardroom.

"Most board evaluation is done in executive sessions [closed to the public and the press], and in most states is mandated by statute," according to AASA's 2000 Study of the American School Superintendency. "Often the evaluation is given at the same meeting that the superintendent's contract is rolled forward or a new contract is offered. Seldom are the results of the evaluation made public except in a general manner.

"What I hear most of the time from superintendents is that their board spends all of five to 10 minutes per year on the evaluation," says Joe Rudnicki, superintendent in Sunnyvale, Calif. "I believe this is very dangerous and that leads to serious trouble. How can a board believe in a process that has been so cursory for so long?"

Test Connections

New surveys of superintendents show substantial use of evaluations, with a growing interest in tying the assessment to state learning goals. But there are also some surprisingly negative feelings on the part of many superintendents toward the process. Many believe their evaluations often fail to measure some matters, such as conflicting philosophies, that can do serious harm to relations between superintendents and their boards.

AASA's 2000 Study of the American School Superintendency, based on an usually large sample of 2,236 superintendents, shows that 80.3 percent of superintendents nationwide are evaluated annually and 12 percent more often than that.

The most common reason for evaluating a superintendent's job performance, according to the survey, was to establish systematic accountability (51 percent), followed by the need to assess performance on state standards (30.7 percent) and the need to comply with board policy (28.4 percent).

Those results are predictable in the new era of accountability. Superintendent evaluations as an annual ritual are moving toward stronger connections with the influential, high-stakes achievement tests that states are using to decide which school districts are in need of more outside help. But a smaller survey of districts in three important states suggests some unspoken resentment on the part of the superintendents being evaluated.

The study of 138 superintendents in Illinois, Missouri and Texas, published in the Match 2000 issue of Educational Research Quarterly, was conducted by Patricia Cleveland, superintendent in Calhoun, Mo., and professors at University of Missouri, Ball State University and Texas A&M. Nearly all the superintendents surveyed said they had an annual evaluation, and most indicated it was of the traditional sort. Only six percent said they had 360-degree evaluation.

But when asked to speculate on the focus of their board's evaluation efforts, 52 percent said "it had a significant influence in identifying grounds for termination," says one of the study's co-authors, George J. Petersen, associate professor of educational leadership and policy analysis at the University of Missouri-Columbia.

Further buttressing the ill feeling about the process, he noted, 58 percent agreed or strongly agreed that the evaluation has a significant influence in identifying superintendent weaknesses. Only 66 percent thought the evaluation process was fair and less than half thought it was designed to identify their administrative strengths.

Petersen says he and his co-authors were surprised by that reaction, as were some other experts. Warren, the executive recruiter in Illinois who also works with districts on their evaluation processes, says most of the boards he has dealt with have healthy assessment systems " which the evaluation instrument is used as a growth tool, not as a reason to get rid of the superintendent."

Sour Moments

Yet some superintendents say the evaluation process is often too weak to bear the occasionally heavy load of distrust and may, in some cases, make the situation worse.

"I believe that a superintendent must do what he or she can to tease out the perceived difficulties and talk with the board about those areas of need," says Rudnicki, the superintendent in Sunnyvale, Calif. "Too often I'm afraid that the superintendent knows he or she has no control over whatever the interfering factors are, but a board member believes the superintendent does have control. The real problem isn't uncovered because they raced through the evaluation and didn't communicate the difficulties."

Better and longer evaluations, Rudnicki says, are a way of "resolving the tension between boards and the superintendent, but whoever believes that that alone will solve the problem is in La-La Land."

Although there appears to be no research on the matter, many superintendents are convinced that good evaluations do not protect them from sour moments with their boards. Martin Hanley, superintendent in Carver, Mass., says while working with other Massachusetts administrators to develop an evaluation model he found that "job performance did not seem to be related to longevity in a position."

"It didn't seem to matter what type of evaluation was used," Hanley says. "You could have a superintendent that got a satisfactory or good evaluation and then was not rehired. You could have superintendents who had no evaluation at all or had many 'needs improvement comments made about him or her and still had a great relationship with the school committee."

Communication with board members is vital, superintendents say, and can smooth out rough spots in the evaluation process.

Doyle F. Scott, superintendent of the East Marshall Community School District in Gilman, Iowa, says one of the goals on his evaluation plan was to make 144 visits to the four schools in his district each year. "One of the board members had a concern that I wasn't spending as much time as I should in the elementary schools," he says. When Scott took the time to explain, the board member realized more than half of his visits were to those schools.

Whatever the superintendent does, says Warren, who was superintendent in two suburban Chicago school districts, ''sometimes you have members of the board of education that have an individual agenda that they want pushed, and if that agenda is not satisfactory, then the data they are looking at may decide whether to keep the superintendent or not keep the superintendent." Such circumstances, he adds, "I would find disconcerting. Sometimes it becomes personal."

Eleanor Smalley, superintendent in the 1,900-student Clarke County, Va., district, says it is rare that superintendents with good evaluations are fired. Instead, if they clash with their boards, "it is more that the person d decides to leave. It happens a lot."

Personal or ideologica1 clashes with previous boards are common enough not to be disabling, but the fact that hiring and firing decisions are made without reference to the usual evaluation criteria creates more distrust about the worthiness of the assessment process.

Some evaluation experts think the system would work better if the assessment forms forced school board members to identify areas where they thought the superintendent was weak. That is common in regular employee evaluations, but many school administrators do not think it would work at the superintendent level.

Linda Kelly, superintendent in New Rochelle, N.Y., says she tells her staff members where they need to improve. She has no trouble doing that, she notes, "because I work with them every single day." Board members see her only twice a month. They would be hampered, she says, by that lack of frequent contact and the difficulties of reaching agreement among themselves on her alleged shortcomings.

Common Criteria 

The feeling is widespread among superintendents that the assessment process will be driven in the future by how well the students in the district do on state-mandated tests, especially as those tests dictate decisions about promotion in grade and graduation. Many districts are talking about tying bonuses for principals and teachers to test scores. Why not rate their superintendents that way?

Some state legislatures already have decreed that school boards cannot sign multiyear contracts with superintendents unless they tie compensation and contract renewal to gains in student achievement. Anthony Scariano, principal partner of a Chicago law firm and a board member of the National School Boards Association's Council of School Attorneys, says this is the law in Illinois and is spreading to other parts of the country.

And yet school boards so far seem reluctant to consider test scores in rating their superintendents unless forced to do so. Experts think this may be explained in part by a strong feeling among school personnel, including board members, that if their students' scores are high, they can't go much higher, and if they are low, it is socioeconomic and not school circumstances that are to blame.

In the survey of superintendents in Illinois, Missouri and Texas, student performance failed to make the list of the five most common evaluation criteria used for measuring school system executives. Instead, the top five were: (1) Board relations (93 percent); (2) Fiscal management (89 percent); (3) Community relations (86 percent); (4) Facilities (75 percent); and (5) Personal qualities (75 percent). Student performance placed seventh (64 percent).

In school districts whose students score high, there is little talk of evaluating superintendents a how welt students are doing. "My evaluation does not have a single question about raising test scores," says Costa in East Longmeadow, Mass., an affluent community where student cores are already among the best in the state. "I would have a huge issue with the district if it did that. They could offer me $100,000 more as a stipend for raising the scores and I still could not do any more than I am doing now."

Many other school districts do not enjoy such high marks on standardized measures, but few superintendents have been punished for that. These are often urban superintendents who rarely stay in their jobs long enough to be blamed for changes in scores. And despite the demands from governors and legislators for higher achievement, most educators do not believe schools an do much about low scores without smaller class sizes and other changes the districts often cannot afford.

Scariano, a veteran school attorney in Chicago, shares the belief that superintendent contracts should not be tied to student performance because there are so many variables our of their control, but the state laws forcing this change are so new it will be years before the effects are seen, he says. Some superintendents sign such contracts "because they have such big egos they think they can make it work," he adds.

Politicians and editorial writers frequently suggest that superintendents be evaluated based on student performance, but actual evaluation practice "has not caught up with the political desire," says James E. Walter, associate professor of education at the University of Missouri-St. Louis and until recently a member of a school board in St. Charles County, Mo. "Many administrators feel you cannot control student performance on tests, and they have a valid concern."

Superintendents do many things that may affect scores, however, and those should be part of a superintendent s evaluation, Costa says. Massachusetts has produced a plan for using new instructional technology, such as computers and the Internet. "One of my goals for both staff and students is improving technology to help students perform better on the state tests," he says.

Slow Growth

Superintendents and experts appear to agree that the trend toward more detailed evaluations and more attention to student performance will proceed gradually. Superintendents are as much of a force behind the change as school boards are, but many of those involved in the process say it takes a great deal of time and motivation to deepen the evaluations.

In Clarke County, Va., superintendent Smalley sought more detail last May by jumping the gun on her own evaluation process. She told her staff she would be sending them more detailed questions in the fall, but "I am particularly interested in improving my performance and the function of the School Board Office over the summer. If you can give me some feedback, I will use it to work on my continuous growth this summer and try to better serve you next year.

The spring questionnaire was just eight questions, which she kept to one page so as not to overburden her already busy staffers. She asked them to list areas where improvements would make her more effective. She also asked: "Please list two things that hindered students' success this year" and "Please list any overall suggestions for the improvement of the school system."

In Lexington, Mass., Ruane insists on submitting her own evaluation each year. She focuses on what she can do to buttress teacher effectiveness and student achievement. "I happen to believe in the shift from an entitlement culture," in which school employees assume whatever they are doing is fine, "to a performance culture," in which results count, "but that tends to be a pretty painful transition," she says.

Will Lexington's school committee do another 360-degree evaluation this year with dozens of interviews and pages of analysis? Elberger, the former school committee member, says "energy is an issue. It is extremely labor intensive."

Ruane says she liked much of what she saw in last year's detailed evaluation, but "I think that, having gone through it, it is certainly my interest to have another conversation with the board about it before we do it just like that again."

Jay Mathews is an education writer with The Washington Post. E-mail: [e-mail]

AAA Survey Findings: The State of Superintendent Evaluation
Reasons a School Board Evaluates the Superintendent
Document dissatisfaction             5.3%
Point out strengths                  7.8%
To determine salary                    8%
Identify areas needing improvement  21.7%
Establish performance goals         24.5%
Comply with board policy            28.4%
Assess performance with standards   30.7%
Periodic/systematic accountability  51%
Other                                4.3%
Note: Respondents were able to give multiple answers.
Note: Table made from pie chart
Superintendents' Most Recent Job Rating
Not evaluated  5.2%
Below average  0.5%
Average        2.6%
Good          22.6%
Excellent     69.1%
Note: Table made from pie chart
Board's Primary Expectation of Superintendent
Leader of reform     2.8%
Managerial leader   36.4%
Political leader    12.7%
Educational leader  40.1%
Other                8%
Source: The 2000 Study of the American School Superintendency
Note: Table made from pie chart

Why the Evaluation Instrument May Not Matter

PAUL M. HEWITT

As soon as I picked up the phone I could tell that Don (a superintendent colleague whose true identity I'm withholding) was very upset. He had just experienced a highly frustrating board meeting attended by more than 50 people, a dozen of whom were speaking out against his contract renewal.

Don's evaluation by his five-member board had been poor, and he was flustered. "I've done everything they've asked. Our curriculum is organized. We passed a facilities bond with a 70 percent yes vote. Our test scores are up. Our schools are clean and well-maintained. I don't know how I could do a better job."

From every indication I have, Don was a highly competent district superintendent who had poured his life into the job. Yet his school board rated him poorly and would not support him against community pressure.

Another superintendent I've known--I'll call him George to prevent embarrassment--is a wholly different story.

George was a mediocre superintendent. His staff was always bailing him out of impulsive decisions and trying to protect him from himself. After discovering that George had falsified information on his job application, the board, instead of automatically dismissing him, met with the teacher's union to figure out how to keep him in his job. Later it was discovered George had committed more serious offenses, the kind that usually sends one to jail. Again the board looked for a way to keep him in office, this time unsuccessfully.

What is the difference between George and Don? George, a good-natured fellow who didn't rock the boat, was "liked." Don, unafraid to make significant changes that weren't always popular, was not.

Unrealistic Premise

Over the years, I've accumulated numerous evaluation instruments for superintendents. Each year, younger, less-experienced colleagues call me, asking to have copies of any evaluation instruments I might have. Their supposition is, "if I have an objective evaluation instrument that measures my clearly defined goals, the board will have to give me a positive evaluation."

This premise is predicated on the idea that people will evaluate you on the job you do, and those extraneous political factors over which you have no control will not enter into the evaluation process. This is a logical assumption and in a perfect world it would be true. Unfortunately, superintendents don't work in a perfect world.

After 14 years as a superintendent, I've drawn certain conclusions about superintendent job evaluations by analyzing my own experiences and learning from friends and acquaintances. As Yogi Berra said, "You can observe a lot by watching."

Whether we want to accept it or not, many evaluations by governing boards are based on emotion and feelings.

In Dan's circumstance, he was doing a great job. From an objective standpoint, he had done everything any board could ask. Unfortunately, they just didn't like him personally. He was not a good fit for the culture of the school district and community. He was an outsider who could never become an insider. No matter what he did, people found fault with it. He just couldn't win.

George, on the other hand, was well-liked. He fit the culture. He didn't try to make significant changes or reforms. He was personable and constantly told people they were doing a great job, even if in his heart he knew his accolades were not deserved. Even if George made a mistake, it would be rationalized away because everyone knew that George was a good guy who. was trying to do the right thing.

Emotions clouded the evaluation process in both cases. When the superintendent's personality' isn't liked, minimal problems become major problems on the evaluation instrument and in the community. When the superintendent is liked, major problems are minimized as board members and community members look beyond the problems to rationalize a positive evaluation.

Personality Reigns

As superintendents, we've been conditioned to focus on clearly defined goals, on change that will make our school districts exemplary and truly benefit children. Unfortunately, I can name dozens of my peers who did great" jobs focusing on kids and improving their districts, but who were released from their contracts (a euphemism for being fired).

I am not trying to minimize the importance of the evaluation process and the value it has in focusing the superintendent on the needs of the school system. However, whether we like it or not, our security on the job often depends more on whether "they like us than on how well we can document our accomplishments on an objective evaluation instrument.

If you disagree and feel superintendents should charge forward to implement dramatic educational change regardless of the human relations element, remember what one of my mentors told me years ago. He said, "Paul, you can't make changes and implement programs that are good for kids if you don't have a job."

Paul Hewitt is superintendent of the Mother Lode School District, 3783 Formi Road, Placerville, Calif. 95667.