Evaluating administrators with portfolios: principals report mostly positive experiences when used as part of a performance review.

Seven years ago when Andy Dotson became principal of Phelps High School in Pike County, Ky., the school was the lowest-performing high school in the district and the fifth lowest in the state. As of last spring, Phelps is the second-highest performing high school in the district and close to being in the top third of all high schools in Kentucky.

All along the way, Dotson has used a portfolio of his work to help guide his own professional development and ensure his superintendent had a complete understanding of the work he was doing.

"Originally I kept a portfolio when I went through the principal internship program," Dotson says. "Since then it's been pretty much a continual process. As I meet the standards, I document it. At the end of the year I can look at what I've done and see if there is anything I need to work on."

Great Possibilities

Dotson's notable experience notwithstanding, the use of portfolios to evaluate the performance of administrators lags far behind their use among classroom teachers. In most parts of the country, evaluation of principals and administrators remains largely the same as it has been for decades--which is to say minimal.

Simple checklists, one-shot interviews, brief site visits and narrative evaluations remain widespread as the tools of assessment. In many school districts, the evaluation includes little or no face-to-face contact, and the principal simply gets his or her evaluation in the mail, leading one researcher to describe them as "infrequent, late, unhelpful and largely a source of administrative bother." In most cases, the performance reviews fail to result in improved leadership. "Little is learned and not much happens," according to a 2003 ERIC report on the evaluation of administrators.

To proponents, the use of portfolios for evaluating administrators offers the promise of a more valid, reliable and authentic form of assessing professional performance. In theory, portfolios promise a much more comprehensive, nuanced and effective evaluation process and offer the possibility of showcasing a broad range of skills and accomplishments that have been demonstrated throughout an entire year or more. In addition, portfolios provide an opportunity for ongoing reflection and continuous improvement, the elusive goal of nearly all school improvement efforts.

Portfolios probably are used most commonly for career advancement by administrators, a distinctly different purpose, according to portfolio expert Genevieve Brown, dean of the college of education at Sam Houston State University in Huntsville, Texas. Brown cites Louisiana, New York, Nevada, North Carolina, Texas and Virginia as states where principals regularly develop portfolios on their own as part of the job search process. In such places, Brown says, "They don't interview people if they do not have a portfolio with them."

Portfolios also are used for professional development purposes and principal training. For example, portfolios are a key part of the Principal Leadership Institute at University of California at Berkeley, where they are used to help candidates reflect on what they have learned and applied while going through the courses and training. Aspiring principals in Chicago now have to develop portfolios to demonstrate their leadership capabilities before they can be hired.

The least common use of portfolios is for evaluation, according to Brown and other authorities, who say a few districts adopted portfolio-based evaluations on their own. Districts in New York, Pennsylvania and Texas have implemented portfolio-based evaluations for a wide range of administrators, including assistant principals and directors of transportation, according to Brown.

In some districts, even the superintendent is evaluated in part through a portfolio. This is happening in Texas, Brown says, largely at the request of the superintendents. "It gives them ownership over their own evaluation," she says. "So many of our superintendents have asked their boards to please let them use a portfolio evaluation process because it enables them to have more of a formative evaluation."

For this reason, the impact of using portfolios to evaluate school administrators is not yet entirely clear, and their relatively recent appearance on the education scene makes them as yet unproven. Those who have experienced and studied the trend suggest that portfolios are a promising, if difficult, approach to improving performance. Their findings and experience offer helpful insights and guidance for others who are searching for better ways to evaluate principals and administrators.

In most cases, administrator portfolios are housed in three-ring, loose-leaf binders, though some are contained on CD-ROMS or on the Internet. Like the more common teacher portfolio, their organization and contents vary widely by school district and individual, but they often contain reference letters, resumes, copies of certifications, articles written by or about the principal, memos and other forms of written communication. Lists of conferences, PowerPoint presentations and awards, if any, also would be included.

Some portfolios also contain school report cards and student achievement data and evidence of work not visible in a school walkthrough. What's most important, say Brown and other experts, are the written explanations and reflections composed by the administrators to highlight their thought process and analyze the impact of their actions.

Positive Experiences

Those who have used portfolios for evaluation or have been evaluated in part through a portfolio generally report positive experiences.

For those being evaluated with portfolios, the portfolio element can provide an opportunity to document achievements, make sure that nothing important is missed, highlight accomplishments that might not otherwise come to light and even promote a certain amount of self-reflection.

For principals like Dotson, the main value of the portfolio is that it documented what he had actually accomplished and ensured there was no doubt about what he said he did on behalf of the school, which enrolls 453 students in grades 7-12.

"If I told you I was 6-foot-tall, blond and blue-eyed, you'd have to believe me," says Dotson. "If a person was going to evaluate me based on talking to me, I could sell myself to them rather easily. The evaluation wouldn't be a true evaluation of what I've done. I can say I had an open house with 200 people there, but it's another thing if I can show him that with the sign-in sheet and with the agenda or with a picture."

Initially, Doston's portfolio was required as part of his participation in Kentucky's statewide principal internship program for newcomers. "It basically included everything that I did," says Dotson. His mentor, his teachers and the program coordinators all reviewed the portfolio during his internship, using it as a way to monitor his progress. Since then, he has maintained a portfolio that he has submitted as part of his yearly evaluation even though it's not required.

Dotson says his portfolio now serves mainly "as a reflection piece." When he notices he doesn't have much to include in a section of the portfolio, he says it makes him reconsider some responsibility or assignment he's been ignoring or overlooking. "Maybe I'm not doing a lot in that area and I should do more."

Frank Buck, principal of Graham Elementary School in Talladega, Ala., has gone through two portfolio-based evaluation cycles. Overall, he considers his experiences to be positive.

Buck believes the portfolio tends to bring to the surface little-noticed success. "These are not things that would have come up during the course of a normal conversation or been naturally observed from being around," he says, citing his use of a memo to replace weekly faculty meetings as one example. The portfolio gives his superintendent a more wholistic view of his performance, Buck says. "This gives him the whole ball of wax."

In fact, Buck believes that portfolios are much more fair than interviews. "A lot of people talk a good game," he says, "but this portfolio is not just talk, it's not just fluff. You have to put your hands on examples of what you claim," he says. "If your student handbook is full of errors or incomplete, that is clear."

Portfolios are used statewide in Alabama for school-level administrator evaluations, according to Buck, either annually or every three years, depending on the individual's tenure status.

Reducing Subjectivity

For school districts that are adopting or planning to adopt portfolio-based evaluation systems for administrators, the portfolios are sometimes part of an effort to make the annual performance review a two-way street.

"Too often, evaluation is something that's done to you," says Patty Fox, evaluation coordinator for the 62,000-student Greenville County, S.C., school system, which will be evaluating principals with portfolios starting this year, with the hope of extending portfolio use to the central office in the near future. "We wanted to give principals a say in their own evaluation."

Those in charge of conducting evaluations also highlight the fairness of the portfolios compared to traditional methods.

"I think it's more objective and less subjective," says Sue Colton, director of leadership development for the Broward County, Fla., Public Schools, where the use of portfolios for evaluation has expanded over the past three years. "With a portfolio you have hard data and you have documentation--not only a descriptor and a process but also the bottom-line results. It leaves that element of the subjective out of it. The data tell the story."

Portfolios also add a real-world element that is hard to replace without spending substantial time in a school or district office. "Portfolios have definitely added value to the evaluation process," says Stephen Nowlin, associate professor at Jacksonville State University in Jacksonville, Ala., a former superintendent who is a state-approved evaluator of superintendents in Alabama, where portfolios are used for all school- and district-based administrators. "One of the principal benefits is that portfolios require actual working papers, which can't really be made up."

Another benefit, according to Nowlin, is that the portfolios require superintendents to explain what their role is in each of the documents they include, as well as whether the outcome was successful. With the help of the outside evaluator, local school board members review the superintendents' portfolios, as well as the results of structured interviews and other feedback.

Even when the supervisor is working in the same building, a portfolio can help ensure a broad, thorough evaluation. "I have my assistant superintendent's portfolio right here in front of me," says Carol Beers, superintendent of the 9,000-student Williamsburg-James City County School District in Virginia, where portfolios used to be required annually and are still used voluntarily by many administrators.

Beers finds it worthwhile to have the portfolio to look at along with the traditional performance checklists of her principals and central-office administrators. Even without the state requirement to compile them every year--a process that the school district deemed too time-consuming for some--many administrators like having them as part of their evaluation process.

In fact, the documents included in a portfolio ultimately may have less importance than the thinking and writing that goes into explaining their importance. The reflection may be as important if not more so than the items that are included.

"The main thing we stress is not the artifacts but the reflection piece," says Brown, whose book on using portfolios for evaluating administrators includes an entire chapter on the importance of self-reflection. "Reflection improves practice and thus schools. It's the reflection that makes the difference." Her co-author, education professor Eileen Irby, echoes this sentiment: "It's not so much the portfolio itself. It is the process of developing a portfolio and the thinking."

Preventing Misuse

For all of these benefits, it remains clear portfolios still can be misused or superficial or unnecessarily time-consuming, just like any other form of evaluation. A portfolio no more represents proof that something was done or done well or had a positive outcome than an oral response in an interview or the results of a survey. And portfolios are just as susceptible to being manipulated or gamed as any other form of evaluation.

Thinking carefully about how much time portfolios take to put together and to evaluate is a key consideration, according to those who have experience using them.

"When we first developed the portfolio system, it was annual," says Beers, whose district found that requiring new portfolios every year was an unrealistic expectation for most principals and ignored differences in how much evaluation is needed for proven and novice school leaders, as well as among principals and other administrators. "It became too much for them to compile right at the end of the year."

Principals tend to agree. "It's time consuming to put together," says Buck, principal of Graham Elementary School in Talladega, Ala., of the 11 sections required in his portfolio and the giant three-ring binder it takes to hold all of the material. "For each of those entries there has to be a page that describes what audience this particular artifact is written for, why that is a good example." Buck figures his yearly portfolio includes more than 30 pages of new text and reflection.

For school districts or states trying to move to online portfolios, the process is especially challenging. "Moving from paper to technology is painful," says Colton, leadership development director in Broward County, Fla.

Her school district is moving to electronic portfolios for evaluating administrators who want to qualify for a salary bonus for excellence. To do so, they must submit a portfolio documenting their work in electronic form. To assist the process, the school district developed a web-based format allowing candidates to respond to questions, write essays and upload materials.

One way to help ensure a portfolio program is not overly time-consuming is to roll it out slowly rather than all at once, refining it along the way. In Broward County, the district leadership decided not to require the portfolio for all administrators at first. Instead, the portfolios are being piloted for two years with a subset of fewer than 100 administrators who wanted the top recognition level.

In Greenville, S.C., the district involved principals of various types of schools in developing the new portfolio evaluation system that is being implemented this year so that the portfolios would seem reasonable and useful to the school leaders. Roughly 100 principals will be evaluated using portfolios this year as part of a three-year cycle.

To be sure, integrating portfolios into the evaluation process rather than relying on them exclusively is particularly important. Portfolios count as part of evaluations in the Talladega, Ala., system, along with onsite observations, a structured interview and a written evaluation by the superintendent conducted every three years for all but the newest principals.

But the portfolio is not necessarily a make-or-break aspect of the performance review. "No one instrument solely determines the outcome of the evaluation," says Talladega, Ala., Superintendent Leonard Messer, who not only uses portfolios to evaluate his principals but is evaluated by his school board in part through a portfolio. "The portfolio does force principals to become more aware of what they're doing, but it's the supervisor's review form that is more important."

The Glitz Factor

Clear guidelines and specific requirements are also key elements of a successful portfolio program.

"You've got some clearcut guidelines about what you want presented," says John Krolewski, principal of South Bendle Elementary School in Bendle, Mich. "Make sure that your guidelines are pretty specific so that you don't have people worried that some people can put together packages better than others."

According to Krolewski, guidelines ensure glitzy packaging isn't the key feature of portfolios. It also prevents the "science fair" phenomenon, where portfolios escalate into three-D models and overly elaborate PowerPoint presentations.

"I think we have to be careful not to let the portfolios become a glorified scrapbook," says Messer, describing the four portfolios in front of him and the challenge for evaluators in telling the difference between substance and show. "One person did spend a lot of time on the graphic arts in their scrapbook," he says. "One is plain Jane but probably has a whole lot more body to it."

Developing a clear rubric is one key way to structure a portfolio and help ensure the fairness and reliability of the portfolio evaluation process, according to several sources. "That way, right up front, everybody knows what it's going to take to get a 4 on that particular goal," says Brown, the book author on portfolio use.

Emphasizing substance is another key. "Last year, a lot of it was a lot of fluff," says Barbara Pullease, who coordinates the Pathways to Administrative Leadership program in Broward County, Fla., about what she observed in principals' portfolios. "There were pictures of students in the cafeteria with no explanation. It wasn't the meat," she says.

To deal with the lack of substance, new requirements during the past year clarified that every item included in an administrator's portfolio needed to be there for a reason and that reason had to be explained in writing.

Finally, to be used effectively, portfolios must address the concern that they do not accurately represent the principals' own work. "In theory, anybody could have somebody put a portfolio together for them," says Messer, who evaluated four principals in Talladega last year using portfolios as part of the process.

The superintendent admits he is not certain in every case that other school staff did not help compile or even write parts of the portfolios. "In one case, I learned that some of the portfolio was possibly done by one of the staff members. I really wasn't sure if that individual had written it himself. I've got to find a way to weed that out."

Unclear Impact

The impact and usefulness of portfolios varies widely, according to those who have experience with them. Even the heartiest proponents concede there is no hard data showing that portfolios make for better superintendents or principals nor any obvious consensus among those who have personal experiences using them for evaluation purposes.

Buck, an elementary school principal for eight years, credits portfolios for promoting self-reflection. "Having to do a portfolio helped me to look at the whole total job," he says. "When I start to pull examples and put my hands on things that I have done and things that I have produced that prove those things, it makes me stop and think is this really an area of strength?"

Beers, a superintendent, sees the use of portfolios in her Virginia district contributing to more reflective administrators. "By reflecting, that's how you improve. But is there a straight-line effect on performance? Absolutely not. Some portfolios are more thoughtful than others," she says.

But Nancy Moga, who heads Callaghan Elementary School in Covington, Va., has yet to be convinced her portfolio makes her a better principal. "It's just another way to demonstrate how I do my job."

Moga has been compiling a portfolio voluntarily for the past year at her school in rural Alleghany County, a district with 2,880 students.

"It really doesn't make me do things differently," says Campbell, assistant superintendent for the 2,900-student Alleghany County, Va, public schools, who keeps a portfolio as part of her own professional development and as a show of cooperation with district teachers, who are evaluated in part through portfolios. "I just do my job the way I think it should be done."

Still, nearly everyone agrees that portfolios can be an excellent tool for showing supervisors what an educator has been up to all year. Campbell's superintendent, in reading her portfolio, admired the revamped technology plan she devised after a series of meetings throughout the year. "I think my superintendent was surprised and impressed," she says.

And portfolios can help keep even the most experienced administrators on their toes.

"Having been in this for so long, it does sort of help me keep focused," says Messer, a superintendent for 12 years and an educator for 34 years, who is evaluated through portfolios by an independent evaluator appointed by his state education agency. Portfolios are a good reminder to new and veteran administrators that everything they send out is going to become a potential part of their portfolios, he says. "it does help you keep focused on the paper you're putting out."

Alexander Russo is a free-lance education writer in Chicago. E-mail: [e-mail]