Credible Evaluation: Not Yet State-of-the-Art.

The authors' nationwide study of superintendent assessment finds weak criteria and few comprehensive models

The centerpiece of the current reform wave in education is the setting of standards with state legislatures and state boards of education mandating standards for curriculum, school accreditation and student learning. As a result, educators at every level, from classroom teachers to superintendents, are feeling the tight grip of accountability.

Several years ago, legislation was enacted in Illinois linking superintendent contracts to student performance. In 1999, the Virginia General Assembly took standards setting to the next level when it enacted legislation requiring uniform performance standards be used by local school boards in evaluating superintendents, other administrators and teachers. One mandatory criterion for evaluation captured our attention as Virginia educators: improving student achievement.

The inclusion of "improving student academic progress," as the legislature phrased it, as a mandatory standard of superintendent performance highlights the critical role of the district's chief executive in curriculum planning and development, instructional leadership and, ultimately, student performance results. We wondered if the action of the Virginia legislature was unique or trend-setting. How does it compare to what is happening in the rest of the country?

About a year ago, these questions prompted our investigation of the current state of superintendent evaluation nationwide. We collected and analyzed superintendent evaluation procedures, guidelines and instruments issued by state departments of education.

In addition, we compiled documents from state affiliates of AASA and the National School Boards Association that provide recommended frameworks or instruments for superintendent evaluation. However, the professional affiliates in eight states (California, Delaware, Florida, Indiana, Iowa, Kansas, Maine and Nevada) reported they had neither state guidelines nor recommendations on superintendent evaluation for their constituents.

Our national study examined descriptions of evaluation guidelines and/or processes, copies of paradigm evaluation instruments, and copies of state-mandated evaluation procedures (where applicable).

Shared Standards

It seems only sensible that evaluations of educators, including superintendents, should be grounded in the duties of the particular professional groups. Thus, basing performance assessment on the professional competencies and job duties should be a logical and necessary initial step in designing a superintendent evaluation system.

Unfortunately, attempts to define the role of the superintendent and establish priorities for performance by local or state boards often fail to include many dimensions of the role. Local school boards traditionally have defined the responsibilities of the superintendent in terms of a job description--a generalized overview that often is loosely related to actual job responsibilities and even more loosely connected to the superintendent's performance evaluation. Most superintendents have a job description, but in AASA's 2000 Study of the American School Superintendency only 50.2 percent said they are evaluated according to the criteria in the job description.

In an attempt to define the profession of the superintendency, AASA established a commission in 1992 that developed a set of eight profess anal standards and a corresponding set of competencies. When coupled with defined professional responsibilities, the standards can serve as a framework for defining the role of superintendent and serve as a basis for meaningful evaluation (see Table 1, page 21).

The superintendent performance competencies embedded in the evaluation materials collected in our 1999 national study were compared to the AASA's "Professional Standards for the Superintendency." When language in a state document wasn't identical but clearly fell under a standard, it was considered to be a partial match. When language in state performance indicators mirrored an AASA standard, we counted it as a complete match. If no language met the two criteria, the indicator was not counted as a match in this analysis. Only the recommended evaluation materials from Hawaii, Oregon and Texas fully matched all of the AASA professional standards.

The AASA standards are reflected in materials gathered from all states. However, the manner in which they are incorporated varies greatly from state to state. Organizational management was the most common standard reflected in state evaluation information (88 percent). The least common standard reflected was values and ethics of leadership (26 percent). More recently, AASA's 2000 study of the superintendency found that the criteria most frequently used to evaluate superintendents were "periodic/systemic accountability" and "assessing performance of district attempts to meet standards." Clearly, the national trend of high-stakes testing has had an impact on how superintendents are being evaluated.

Why Evaluate?

Legislation in most states makes boards legally responsible to evaluate their superintendent. The evaluation process can be a valuable tool in defining expectations, enhancing communication, identifying and prioritizing district goals, and holding the superintendent and the board of education accountable.

Moreover, evaluations can provide superintendents with an opportunity to assess the board's satisfaction with their performance. Evaluations also give board members an opportunity to assess the superintendent's job satisfaction.

School boards cite a broad range of benefits of superintendent performance evaluation, the most common of which is to direct and encourage school improvement. Boards also commonly cite improvement in communication, planning, budgeting and relations with the board.

Other benefits of a structured evaluation system include:

* clarifying roles of the superintendent and board members;

* communicating expectations;

* enhancing professional development;

* providing accountability;

* fulfilling legal requirements; and

* making employment decisions.

In AASA's 2000 Study of the American School Superintendency, superintendents reported the major purposes of their evaluation as setting performance goals and ensuring systematic accountability. School boards and superintendents tend to agree on the benefits of performance evaluation.

Evaluation Models

The materials on superintendent evaluation collected from the states in our study were compared with evaluation models used prominently in business. The three most common evaluation models in use focus on global judgment, judgment driven by criteria or judgment driven by data.

Global judgment consists of several prominent evaluation practices, including school board judgment, descriptive narrative reports, formative exchanges about performance and/or stakeholder evaluations.

Judgment driven by criteria may consist of printed rating forms, report cards, management by objectives, performance contracting and/or duties-based evaluation.

Judgment driven by data includes superintendent portfolios, student-outcome measures and school and district accreditation.

Most AASA and NSBA state affiliates offered recommended rating forms. Management by objectives and printed rating forms were the most commonly used types of evaluation with 89 percent of the states employing some variation of MBO and 68 percent of the states embedding management by objectives into their performance evaluations.

All models in this analysis contained relative strengths and weaknesses. No perfect superintendent model, policy or practice exists. Nonetheless, the Personnel Evaluation Standards, developed in 1988 by the Joint Committee on Standards for Educational Evaluation, provide guidance for effective superintendent evaluation. Using the standards as a filtering device, models in which strengths overshadowed weaknesses include duties-based evaluation and the use of superintendent portfolios. Unfortunately, our data reveal that only one of the 42 state respondents use portfolios in their superintendent evaluations, while 11 feature duties-based evaluations.

Standards for Evaluation

Despite the recommendations for personnel evaluation standards proposed by the Joint Committee, eight of the AASA and NSBA state affiliates do not have requirements or guidelines for evaluating superintendents. Evaluation materials provided by affiliates in the other states do not meet all of the essential criteria for quality personnel evaluation. These criteria include:

* propriety standards related to legal and ethical considerations;

* utility standards related to the timely, informative and influential nature of the evaluation;

* feasibility standards related to ease of implementation and use; and

* accuracy standards related to technical accuracy of information obtained about the educator's qualifications or performance.

The standard most often absent was accuracy. The accuracy standard requires valid, reliable and systematic data, free from bias. Since school board members are not always trained adequately to complete evaluations to meet this criterion, the fairness of evaluations is a serious issue.

In Alabama, a trained state evaluator actually conducts the evaluation of the superintendent for each board. But this is not common practice. Several states require training of new board members and most state affiliates provide information related to guidelines for good evaluations. Several state school board associations indicated the offer training to school board member , but the frequency and extent of the training was unclear. Because turnover in the ranks of school board member can be high, new members should be trained adequately to evaluate superintendents.

Immediate Implications

The growing use of student scores on state and other standardized tests to compare schools and school districts places school leaders in a fishbowl of public scrutiny. Within this high-stakes environment, the working relationship between a board and the superintendent can deteriorate rapidly.

A downwardly spiraling relationship sometimes is expressed in terms of community dissatisfaction and conflict, which, in turn, results in election of new board members running on a platform to replace the superintendent. In this context, accountability is a double-edged sword that cuts to the heart of two critical issues: fair and unbiased evaluation of superintendent performance and superintendent job security.

Superintendents are the only school district employees not supervised or evaluated by another licensed professional. Yet it is imperative that superintendents be evaluated in a manner that meets all the criteria of good personnel evaluation. This will only happen for superintendents when credible evaluation models are adopted and evaluators receive better training.

Superintendents are in a position to work with their boards to develop an evaluation system that is mutually beneficial. We suggest the following guidelines in that endeavor:

* Know what is expected. Superintendents should insist that their job description accurately describes what they are expected to do.

* Use AASA standards as a framework. The Professional Standards for the Superintendency, while not necessarily intended by the association to be used in performance evaluation, and the corresponding competencies can be adapted and applied to each superintendent's job setting.

* Ensure the evaluation criteria reflect the job description. Performance should be assessed on how well the superintendent meets those expectations.

* Encourage the use of multiple data sources. Superintendents should work with their boards to agree to apply multiple sources of feedback, such as staff input and progress in meeting goals, in their performance evaluations. Superintendents deserve a quality evaluation that meets all of the personnel evaluation standards. Agree on using evaluation data that are valid, reliable, systematic and free from bias.

Michael DiPaola, a former superintendent in New Jersey, is an associate professor of educational policy, planning and leadership at the College of William and Mary, P.O. Box 8795, Williamsburg, Va. 231 87. E-mail: [e-mail]. James Stronge is Heritage Professor of educational policy, planning and leadership at the College of William and Mary.

Table I: AASA Professional Standards

The AASA Commission on Standards for the Superintendency issued a 16-page document in 1992 that described performance standards in eight areas, as follows:

* Leadership and District Culture

* Policy and Governance

* Communications and Community Relations 

* Organizational Management

* Curriculum Planning and Development

* Instructional Management

* Human Resources Management

* Values and Ethics of Leadership.