Problems with subjective in-training evaluations.

To the Editor: How does a program assess the clinical performance of physician trainees? In-training evaluations (ITEs) were created for this purpose. (1) ITEs should be based on training objectives, recent faculty observations, and reflect a fair and equitable process. However, most ITEs are subjective (1) and can thus be subject to many kinds of bias. (2-4) Faculty may also have difficulty in providing negative evaluations--particularly if they have experienced a complaint regarding their past evaluations.

We would like to recount our experience with ITEs. We employed a comprehensive system of training objectives with a six-point grading scale: "unacceptable," "needs to improve," "acceptable," "good," "above average" (better than 60% of residents), and "outstanding" (better than 95% of residents). Initially, individual faculty performed each ITE. A review of 50 consecutive ITEs revealed the majority of evaluations to be "good" or "above average," whereas only rarely was performance deemed "unacceptable" or "in need of improvement." Moreover, we dispensed "outstanding" grades at twice the defined frequency.

There were other problems. Two residents complained that they had been unfairly underrated. One resident, who never had been evaluated at less than "acceptable," was identified by written comments from a single ITE to have major deficiencies that, in retrospect, had been overlooked on prior evaluations.

In an attempt to minimize bias, committees comprised of three faculty performed all further evaluations. With this reform, "acceptable" grades were dispensed about twice as frequently as before, "good" grades were given 30% more frequently, whereas "above average" and "outstanding" grades were granted about half as often. Some faculty, however, expressed the view that "outstanding" grades should be dispensed more liberally, as an encouragement and reward. In addition, about half of the trainees complained that the expectations placed on their performance were unrealistic. Some trainees attempted to have their evaluations upgraded or to avoid rotations with faculty they perceived to be more stringent.

Our system of graded objectives seemed inefficient in identifying residents in need of help. Written and verbal comments offered by individual faculty were often more informative, and despite the predominance of excellent grades, such comments were made frequently. We felt unable to act upon verbal comments alone, as these had been provided without the controls requisite to ITEs.

We encountered two different evaluation approaches. With a laissez-faire approach, above-average evaluations would be dispensed liberally; the national examination boards would then ensure physician competence. Under a more stringent approach, the majority of evaluations would be "satisfactory," and the strengths and weaknesses of individual trainees would be identified and corrected before board examinations.

The stringent approach might best address the educational needs of trainees. But, as we had observed, it was difficult to control mark inflation, and this approach was unpopular. We concluded that it was unrealistic to utilize a subjective grading scale any more discriminating than "pass" or "fail." To circumvent many of the problems we experienced, there was a need for objective, structured clinical examinations to supplement or replace our subjective evaluations.

References

1. Turnbull J, Gray J, MacFadyen J. Improving in-training evaluation programs. J Gen Intern Med 1998;13:317-323.

2. Lacorte MA, Risucci DA. Personality, clinical performance and knowledge in pediatric residents. Med Educ 1993;27:165-169.

3. van der Vleuten CPM, Norman GR, de Graaff E. Pitfalls in the pursuit of objectivity: issues of reliability. Med Educ 1991;25:110-118.

4. Maxim BR, Dielman TE. Dimensionality, internal consistency and inter rater reliability of clinical performance ratings. Med Educ 1987;27:130-137.

Wayne S. Kendal, MD, PHD

Robert MacRae, MD

Paul Dagg, MD

Division of Radiation Oncology, Ottawa Regional Cancer Centre, and Office of Postgraduate Medical Education, University of Ottawa Ottawa, ON, Canada