Medical students' perceptions of feedback in a busy ambulatory setting: a descriptive study using a clinical encounter card.

Background: Residents and medical students have expressed repeated concerns over the years about the inadequate amounts and quality of feedback in the clinical setting. Despite innovative ways to teach the skill of giving feedback, the problem has not been fixed.

Methods: In this study, the author introduced the clinical encounter card to the ambulatory setting for faculty to use as a cue to provide feedback to students. At the end of the 4-week rotation, students anonymously reported on the amount, frequency, and quality of feedback they received.

Results: Students reported that the learning climate for giving/receiving feedback was very good to excellent. They rated the quality and amount of feedback they received from faculty as high on the Likert scale and the frequency just above the mean. Most of the feedback was directed toward knowledge and skills, and there were few reports of demeaning behavior. Feedback was timely, and students reported using the feedback to improve their performance. The clinical encounter card improved feedback to students in a busy ambulatory setting. Whereas the author did not monitor how often the clinical encounter card was used, there were ample cards on each student to provide mid-rotation feedback and summative evaluations. The students rated the process as the best of any clerkship rotation.

Conclusions: The clinical encounter card is an effective tool to enhance feedback in a busy pediatric ambulatory setting. It is not known if these results are generalizable, but readers are encouraged to repeat the study in other settings.

Key Words: ambulatory, feedback, medical students

**********

The lack of adequate amounts of effective feedback in the clinical setting has been identified as a major ongoing problem in medical education. (1-5) This problem persists despite faculty development efforts to train faculty and residents to provide more frequent and effective feedback. (6-8) Whereas numerous authors have suggested ways to improve the feedback process, none of these approaches appears to have "fixed" the problem. (2,9,10) Overall, students have reported less interaction with faculty and residents, and a decrease in the quality of teaching within the managed care environment. (11)

In the current academic environment, multiple faculty oversee medical students a variable amount of time during each ambulatory block rotation, while having other pressing responsibilities: seeing their own patients, participating in other scholarly activities, attending on the wards, performing departmental administrative responsibilities, staffing resident continuity clinics, and teaching other students within the medical school curriculum. Most ambulatory rotations are limited in length, making it difficult for preceptors to develop an ongoing, one-to-one relationship with learners. These circumstances make it more challenging for faculty to provide adequate, timely, and appropriate feedback to medical students.

In the pediatric clerkship at Children's National Medical Center, medical student summary evaluations over the years have consistently documented the lack of faculty and resident feedback. This problem has been without question the number one complaint raised by students in end-of-clerkship evaluations. To improve this process, the author introduced a clinical encounter card (CEC) system 10 years ago in the Children's Health Center (CHC) at Children's National Medical Center (CNMC) as a way to systematize and provide an easy and efficient process to stimulate feedback from faculty to students. As first reported by Rhoton, (12) the CEC was used among a group of anesthesiology residents who had reported similar continuity and feedback problems in working with their attending physicians. Rhoton described a typical academic anesthesiology model, that is, a large number of faculty working different shifts with different residents, making it difficult for any one physician to evaluate residents over time. The anesthesiology model compares with that in medicine and pediatrics because many attending physicians oversee trainees, and few are in the setting every session.

The hypothesis of this study is that student reports of inadequate faculty feedback on the end-of-clerkship evaluation would be significantly reduced. It was further postulated that the CEC could act as a trigger for faculty to provide feedback, regardless of whether the process was initiated by the teacher or the learner.

Materials and Methods

This descriptive study took place in CHC in the 1998 to 1999 academic year (July to May). The clinic comprises 14 attending physicians, equal to 5.4 full-time equivalent attendings, and 2 pediatric nurse practitioners. All of the pediatric residents rotate in CHC for 2 months of block time. This center represents the major primary care site for infants and children at CNMC, with 22,314 patient registrations in 1999. Medical student ratings of faculty teaching in CHC over time have been excellent, that is, 4.45 on a Likert scale of 1 to 5, with 5 being outstanding. This excellent teaching record has enabled CHC to rank as one of the top teaching sites in the clerkship, this despite the poor faculty track record on feedback. Faculty members have participated intermittently in teaching improvement workshops, which have focused on the educational process in ambulatory pediatrics, including feedback in this setting. However, none of these workshops incorporated the CEC concept as a solution to improve feedback.

Third-year medical students from George Washington University rotate through CHC and other CNMC ambulatory sites as part of their pediatric clerkship ambulatory requirement. Before the start of their clinical clerkship year, the students have an opportunity, on a lottery basis, to choose predetermined linked inpatient and ambulatory sites on the pediatric clerkship. This system ensures that all students experience an ambulatory rotation and spend some time at CNMC. Of a class of 160 students per year, 42 are assigned to CHC, with 30 of these for 4 weeks, and the remaining students for 2 weeks. Before the study, CHC attending physicians had implemented the CEC as a way to give feedback to medical students. The purpose of this study was to document that feedback from the students' perspective.

The faculty-student interaction can be described as follows. After a student-patient interaction, a faculty member overseeing the student completes the CEC by filling in the appropriate checklists, writing narrative comments, and sharing his or her observations with the student, prospectively. Faculty members were encouraged to record patient diagnoses, strengths, and weaknesses of the encounter and student-centered learning objectives. The back of the CEC is blank, to accommodate this information (Figure). Estimated time to complete each card ranged from less than 1 minute to 2 minutes, depending on the extent of the narrative comments.

Students, who had access to blank CECs if faculty did not initiate the process, understood their investment in this process: (1) the CECs provided chronological documentation of their knowledge, skills, and attitudes in CHC; and (2) the cards represented an essential part of their mid-rotation and final evaluations. The subject areas on the CEC include observed history, observed physical examination, oral presentation, written presentation, assessment skill, and management plan.

Measures

Students anonymously completed an exit questionnaire after each 1-month ambulatory experience that focused on their perceptions of the quality and frequency of feedback, the generation of student-initiated learning objectives, mid-rotation feedback, and student self-perceptions of improved performance. Students' perceptions of the quality and quantity of feedback were assessed through the use of a 1 to 5 Likert scale. Students also evaluated whether or not the learning climate was conducive to providing effective feedback: specifically, well-defined goals and objectives, clear expectations, and a trusting environment. (13,14)

[ILLUSTRATION OMITTED]

Feedback was divided into the three educational domains: knowledge, attitudes, and skills. Feedback was also considered to be reinforcing, corrective, and demeaning. Students replied as to whether or not they perceived the feedback they received to be challenging and how often they received compliments, not to be confused with feedback.

None of the faculty was aware of the data collection during the entire study period. There was no attempt to in-dependently measure direct observations of faculty and student behavior in the study or assess the quality of the case presentation or overall interactions; nor did the author examine interpersonal characteristics and how these might have affected student responses. The study predated the need for institutional review board approval and would have been viewed as an educational intervention and not research at the time.

Results

Twenty-seven medical students were assigned to CHC during the 11-month study period and completed the questionnaire, a 100% response rate. Students who spent 2 instead of 4 weeks in CHC were not included in the data. Table 1 summarizes student perceptions of whether or not the learning climate was conducive for effective feedback, namely, defined goals and objectives, clear expectations, and a trusting relationship.

Table 2 summarizes student perceptions of feedback during the study period.

The students rated the amount and quality of feedback that they received from faculty as high on the Likert scale, with the frequency of feedback just above the mean. They stated that faculty directed most of their feedback toward students' knowledge and performance, with less focus on attitudes. The characterization of feedback was divided between reinforcing and corrective comments, with few students reporting demeaning and abusive faculty behavior. Students said that faculty asked about their learning objectives for each patient more often than not and rated the overall use of the CEC as an adjunct to providing feedback as somewhat effective.

All students reported receiving timely feedback after patient and attending interactions. Three quarters of the students reported receiving mid-rotation feedback; 50% believed that the feedback was helpful to their learning. Students reported that they used the feedback to improve their performance, with a mean Likert score of 4.1/5. As part of feedback, faculty encouraged students to develop their own learning objectives in 68% of their interactions. Only four students (15%) said that preceptors never shared feedback written on the CEC on specific patients. Ninety-five percent of the students rated the feedback process as outstanding to above average at CHC, as compared with other clerkships.

There was no attempt to track and document the number of CECs on each student-patient encounter. Although faculty members stated that they did not complete a CEC after each student-patient interaction during the rotation, quantitatively they completed sufficient cards documenting student performance to provide meaningful and ongoing feedback to each student at the mid-point and the end of each rotation. Attending physicians varied, regarding the amount of narrative they wrote on each CEC.

Discussion

Although clinical faculty, students, and residents agree that summary and formative feedback are essential components of the educational process, feedback to learners on their performance remains a major deficiency in the clinical setting despite teacher training in this area. It is unclear why this problem persists, but factors may include poorly defined goals and objectives, unobserved student performance, and the failure of faculty to develop a mutually trusting relationship with learners. (14) Other issues might be fear of hurting the learner, avoidance because faculty are not perfect themselves, and no formal training in this area. A recent study revealed that faculty are quick to reply that they value feedback and provide it all the time to learners, whereas the latter respond that they rarely receive feedback. (15) Perhaps learners do not always recognize feedback when it is given, and coaching them to be active participants in the process can improve their perceptions of feedback. (16)

The purpose of this descriptive study was to determine if faculty feedback in a busy ambulatory setting could be improved through the eyes of the students by using a CEC. In addition, it was anticipated that the CEC would trigger feedback, whether the card was introduced by faculty or students. Rhoton first described the use of a clinical anesthesia system of evaluation (CASE) method that was implemented in an anesthesiology program to "expand the dimensions of the evaluation process and improve validity." (12) As an additional effect, it was also used to improve the feedback process at that institution. In her study, Rhoton demonstrated that feedback improved when anesthesiology faculty, working different shifts with many trainees, used a CEC. Using Rhoton's methodology, Brennan and Norman (17) reported on the use of an encounter card to assess this as a tool for formative evaluation. They reported six resident encounters with faculty and demonstrated a higher correlation of resident behavior documented on the cards with an oral objective structured clinical examination and a multiple-choice examination. Lye et al (18) used the author's CEC model to cue pediatric faculty to provide feedback to students prospectively on each patient the student encountered. Faculty acknowledged that the card was easy to use, and student comments were positive. The CEC appeared less useful in ambulatory areas, where there were only a few faculty staff members and a limited number of patients. Paukert et al (19) used an encounter card system to improve student satisfaction with the feedback process in surgery.

The amount and quality of feedback students reported in this study was gratifying (3.9 to 4.1/5.0) when compared with reports in the literature. The frequency with which students acknowledged receiving feedback was rated at lower numerical values, and, in fact, four students stated that their preceptors never shared feedback from the CECs over the month. This was contrary to the fact that each had multiple CECs documenting his or her interactions and performance with patients. This disparity could be explained by the fact that students do not always recognize when faculty are providing feedback and that faculty verbally related but did not always visually share the contents of the CECs with the students. It was clear from the study results that the CEC represents a vehicle to remind students and faculty of the importance of prospective feedback, which is then used for summary evaluation.

Students reported that the feedback they received was more directed toward their performance and knowledge as compared with their attitudes with patients, attitudes being more difficult to measure. Importantly, students considered feedback they received to be bi-directional, that is, toward encouraging positive behaviors and correcting those behaviors that fell below the expected standards. They perceived that faculty conveyed compliments (eg, "That was a really good presentation") much less frequently than focused, reinforcing, and corrective feedback. In addition, students did not think that faculty were demeaning in how they gave feedback, suggesting the absence of abuse, which students often report in the annual Association of American Medical Colleges graduation questionnaire. (20) They described faculty as challenging, reinforcing the concept that faculty can inspire and stimulate learners without being abusive.

Students reported that faculty feedback often made an impact on their subsequent performance, for example, case presentations, interactions with parents and patients. They also stated that faculty encouraged them, although less often, to develop their own learning objectives after each patient interaction. This reflection-in-action idea is an expectation in CHC, that is, students will identify learning objectives on the basis of their patient interactions, pursue these objectives, and then discuss these with faculty. Anecdotally, this occurs frequently, since this learner-centered exercise is one of the important measures faculty use to assess student performance.

The large majority of students thought that they received timely feedback after their patient interactions, and one half thought that this feedback was helpful. Therefore, even in a busy managed care environment, faculty can succinctly and effectively tell students in what areas they have performed well and what needs fixing.

Effective feedback will not happen if a positive learning climate is not established. In this study, students were confident that they understood the goals and objectives of the ambulatory rotation and knew what was expected of them. In addition, students perceived faculty as reaching out to develop trusting relationships with them, suggesting an optimal environment for learning. We did not attempt to document faculty observations of student performance in this study, although characteristically each student is observed in patient interactions at least 3 to 4 times over the month, with faculty using the brief structured observation technique. (21)

An unexpected outcome of this study was the fact that 95% of the students anonymously rated the feedback process in CHC as above average to outstanding, more effective than other clerkships. This anonymity of student reporting appeared to account for their frank and open responses without fear of retribution. Whereas the results did not prove that the CEC was solely responsible for this high student rating, no other clerkship in the medical school uses this process. Certainly, faculty interest and motivation might have accounted for some of this effect.

Conclusion

There are limitations of the study. Before the introduction of the CEC, there was no baseline measurement of feedback in CHC. As the pediatric clerkship director for 22 years, the author's subjective observations were that the feedback process improved significantly after the introduction of the CEC. A major purpose of this study was to quantify this impact. The variables that appeared to contribute to making the process effective include a faculty who recognize the importance of feedback, are committed to using the CEC as a tool, and who set the ground rules and learning climate for the rotation at each student orientation. Another limitation includes the use of student self-reporting. This process can be inaccurate, although students completed the anonymous questionnaire the last day of each month of their ambulatory rotation, while the experience was fresh in their minds. Videotaping and ongoing faculty observations would have helped to validate student perceptions, but there were limited resources. Finally, since students had a choice in the selection of their rotations on the clerkship, this may have influenced their outlook on the clinical experience. A randomized, controlled trial would have been the ideal study design. Finally, the author is uncertain about the generalizability of the data but would recommend that other ambulatory sites use this process to evaluate the effectiveness of the CEC.

The implications from the study are that the use of a CEC can improve the amount, timeliness, frequency, and quality of feedback to students in a busy ambulatory setting staffed by many faculty. The CEC process, which will require discussion among faculty or possibly some faculty development training, can assist in improving a major deficiency in our educational system.

Table 1. The learning climate

Defined goals objectives       24 (92%)
Clear expectations             19 (73%)
Trusting relationship          25 (96%)

Table 2. Students' perceptions of feedback

Characteristics of feedback        Likert score 1-5

Amount                             4.1 (a)
Quality                            3.9 (a)
Frequency                          3.1 (a)
Re: knowledge                      2.9 (b)
Re: performance                    3.0 (b)
Re: attitudes                      2.0 (b)
Reinforcing                        3.3 (b)
Corrective                         2.9 (b)
Complimentary                      2.3 (b)
Demeaning                          1.1 (b)
Challenging                        3.1 (b)
Impacted performance               4.2 (b)
Learning objectives                2.9 (b)
Rate the process                   3.4 (a)

(a) 1 = poor / 5 = outstanding
(b) 1 = never / 5 = always

Acknowledgments

The author thanks Drs. Jim Perrin and Ben Siegel for their attention to detail and comments about the manuscript.

Accepted April 5, 2004.

Presented in part at the Pediatric Academic Society meeting, 1999.

References

1. Gill DH, Heins M, Jones PB. Perceptions of medical school faculty members and students on clinical clerkship feedback. J Med Educ 1984; 259-856-864.

2. Irby DM. What clinical teachers in medicine need to know. Acad Med 1994;69:333-342.

3. Smith CS, Irby DM. The role of experience and reflection in ambulatory care education. Acad Med 1997;72:32-35.

4. Ferenchick G, Simpson D, Blackman J, et al. Strategies for efficient and effective teaching in the ambulatory care setting. Acad Med 1997;72:277-280.

5. Ende J. Feedback in clinical medical education. JAMA 1983;250:777-781.

6. Skeff KM, Stratos GA, Bergen MR, et al. The Stanford faculty development program: a dissemination approach to faculty development for medical teachers. Teach Learn Med 1992;4:180-187.

7. Anderson VA, Carline JD, Ambruzy DM, Irby DM. Faculty development for ambulatory care education Acad Med 1997;72:1072-1075.

8. Irby DM, Ramsey PG, Gillmore GM, Schaad D. Characteristics of effective clinical teachers of ambulatory care medicine Acad Med 1994; 66:54-55.

9. Henson MG, Little MC. Giving feedback in medical education. J Gen Int Med 1998;13:111-116.

10. Bing-You RG, Greenberg LW, Wiedermann BL, Smith CS. A randomized multicenter trial to improve resident teaching with written feedback. Teach Learn Med 1997;9:10-13.

11. Bragg D, Treat R, Simpson DE. Have clinical teaching effectiveness ratings changed with the Medical College of Wisconsin's entry into the health care marketplace? Acad Med 2000;15(supplement):S59-S61.

12. Rhoton MF A new method to evaluate clinical performance and critical verdicts in anesthesia: quantification of daily comments by teachers. Med Educ 1989;23:280-289.

13. Casbergue J. Role of Faculty Development in Clinical Education, in Morgan MK, Irby DM (eds): Evaluating Clinical Competence in the Health Professions. St Louis, CV Mosby Co. 1978, pp 171-186.

14. Fernald DH, Staudenmaier AC, Tressler CJ, et al. Student perspectives on primary care preceptorships: enhancing the medical student preceptorship learning environment. Teach Learn Med 2001;13:13-20.

15. Smith CS, Francovich C, Gieselman J, Servis M. A broader theoretical model for feedback in ambulatory care. Adv Health Sci Educ 1998;3:133-140.

16. Bing-You RG, Bertsch T, Thompson JA. Coaching medical students in receiving effective feedback. Teach Learn Med 1998;10:228-231.

17. Brennan BG, Norman GR. Use of encounter cards for evaluation of residents in obstetrics. Acad Med 1997;72:S43-S44.

18. Lye P, Bragg D, Simpson D. Improving feedback with a clinical encounter form. Acad Med 1997;72:444-445.

19. Paukert JL, Richards ML, Olney C. An encounter card system for increasing feedback to students. Am J Surg 2002;183:300-304.

20. 2000 Medical Student Graduation Questionnaire. Association of American Medical Colleges, Division of Medical Education.

21. Pituch K, Harris M, Bogdewic S. The brief structured observation: a tool for focused feedback. Acad Med 1999;74:599.

RELATED ARTICLE: Key Points

* The clinical encounter card represents a user-friendly process for faculty to provide feedback to learners in the ambulatory setting.

* Students can be empowered to initiate the feedback process by having them give faculty this card.

Larrie W. Greenberg, MD

From the Office of Faculty Affairs and the Department of Pediatrics, The George Washington University School of Medicine and Health Sciences, Washington, DC.

Reprint requests to Dr. Larrie W. Greenberg, 9109 Fall River Lane, Potomac, MD 20854. Email: [e-mail]