A tool for the assessment of communication skills.

Abstract

While the effectiveness of standardized patients (SPs) used for evaluation and teaching communication skills has repeatedly been demonstrated, there are significant drawbacks to the use of this technique, including lack of standardization across multiple sites and both cost and time at individual sites. This article describes the initial validation study of the Patient Video Interview, an instrument created to serve as a measure of medical student communication skills that overcomes some of the drawbacks of SP-based assessments and that can be used to supplement such assessments.

Introduction

The ability to communicate effectively is crucial to many endeavors, perhaps none more than the relationship between a physician and a patient. Effective communication skills have been shown to be associated with such patient outcomes as (a) medical knowledge, including recall of information, [1] (b) satisfaction, [2] which may lead to better patient compliance, [3] and (c) physiologic outcomes, including regulation of blood sugar and blood pressure. [4] Since the late 1960s, researchers have studied the natural course of communication skill development in medical students and the effects of various programs designed to increase those skills. Today virtually every medical school in the United States and abroad includes interviewing and communication skills as part of its curriculum. Beginning in June, 2004, Step 2 of the United States Medical Licensing Exam (USMLE) will include a Clinical Skills examination. [5] Foreign medical graduates seeking licensure in the U. S. have had to pass such an exam since 1998. [6]

In their attempts to study communication skills, researchers have used a variety of techniques and measurement instruments. The effectiveness of standardized patients (SPs) used for evaluation and teaching has repeatedly been demonstrated. [7] There are, however, significant drawbacks to the use of this technique. These drawbacks include lack of standardization across multiple sites and both cost and time at individual sites. The development and use of multi-institution objective structured clinical exams (OSCEs) utilizing SPs has shown promising early results. [8] Cost, however, remains a significant factor in all assessments employing SPs. The practical significance of this cost is that students are unlikely to experience more than a handful of these valuable experiences during the course of their medical education.

This article describes the development and initial validation study of the Patient Video Interview (PVI), an instrument created to serve as a measure of medical student communication skills that overcomes some of the drawbacks of SP-based assessments and that can be used to supplement such assessments.

The Patient Video Interview

Designed to assess issue recognition and communication process skills, the PVI is a videotape presentation of the cases of two different SPs, each presented in three vignettes. After each vignette, the tape contains timed still screens during which the participants are instructed to (a) write down the exact words they would say next to the patient at that point in the interview, and (b) list what they see as issues in the case. The YF case is that of an adolescent female presenting for a college physical. The EM case is that of an elderly male with progressive end-stage congestive heart failure. The cases were written to represent two ends of the health continuum, but to be parallel in terms of measurement issues. The cases were written by nationally recognized medical experts in the appropriate fields. The two case authors identified the issues present in their cases. Both the cases and the identified issues were reviewed by additional medical experts. Because the cases in the PVI are rich in both medical and psychosocial issues, there are multiple correct answers or approaches to each case. The PVI is 52 minutes in length and is designed to be administered in a group setting. It provides a uniform testing environment that can be used across sites and class years of participants.

A measurement instrument designed to assess communication skills consists of two parts: (a) the instrument or situation that captures the communication, and (b) the scoring system used to describe or rate the communication. Based on logical, analytical, and empirical work described elsewhere in greater detail, [9] the measurement system defined for the PVI was made up of 23 rating scales, 19 of which could be applied to both cases, and 4 of which were applicable only to the case of the dying patient. Of the 19 scales, 6 came from existing systems. Three items were constructed to represent the proportion of statements that asked or requested information, gave information or directed behavior, and established or maintained rapport, corresponding to the three functions of the medical interview as described by Cohen-Cole. [10] Two items were constructed to ascertain the proportion of statements dealing with the psychosocial aspects and the medical and lifestyle aspects of the case. Single items assessed each of the following: (a) the degree to which the interviewer used language that was medical or infrequent in use, (b) the degree to which the interviewer used authoritarian statements, (c) the degree to which the interviewer made negatively judgmental statements, (d) the degree to which the interviewer focused on the patient as agent, (e) the degree to which the interviewer was empathetic, (f) the degree to which the interviewer overtly talked about the patient's feelings, (g) the degree to which the statements were integrated, connected, and coherent, and (h) whether or not the interviewer referred to other professionals by name or by group. The four additional items constructed for the case of the dying patient assessed whether or not the interviewer mentioned or implied dying in a generic way, explicitly acknowledged that EM was dying, referred to treating EM in order to prolong his life, and referred to treating EM medically for relief of his symptoms.

During development of the PVI, issues of both content and construct validity were considered. Content validity was addressed by having the cases written by nationally recognized experts in the appropriate fields. The case of the adolescent female was written by a physician who was board certified both in Pediatrics and in Adolescent Medicine. The case of the dying male was written by a physician who was board certified in Geriatrics and who was named a Faculty Scholar for the Project on Death in America of the Soros Foundation. Each case author identified the issues present in her case. Both the cases and the identified issues were subjected to review by additional physicians. Participant instructions for completing the PVI were also subjected to content validation. They were tested on a variety of participants for issues of clarity, completeness, and the amount of time required to participate. Both the wording of the instructions and the participation times were adjusted until each of the participants felt that they were optimal. The time allotted for writing responses is 3 minutes for the question, "What exact words would you say next to [the patient] Y. F.?" and 4 minutes total for the questions, "What are all of the issues in the case that you can think of?" and "What are your goals for these issues?" These response times allow participants ample time to respond, yet do not give them extensive amounts of time in which to revise their responses. Construct validity was addressed by extensive review of the literature for identification of a set of theoretical concepts relative to medical communication skills and the possible relationships between these concepts.

Unlike content validity and construct validity, which can be considered during the development of the instrument, convergent validity can only be explored once the instrument is complete and has been tested on a population. The study described below was designed to explore the relationship or the degree of overlap between the PVI and other measures of communication skills that are typically collected by medical schools, primarily SP-based assessment in OSCE format. This exploration of relationship was designed to study convergent validity, which may be defined as significant correlations with related measures. [11]

Initial Validation Study: Methods

In their 1998 review of instruments for the assessment of physician-patient communication, Boon and Stewart [12] found that very few instruments had been compared directly with others and opined that such comparison would provide "a better indication of validity." The PVI was compared with two other measures of communication skills in use at the time at one medical school.

Thirty-seven entering first-year students at a New England medical school voluntarily participated in the PVI during orientation. These students constituted 43% of the 86 students entering the medical school at that time. Some months later, these students participated in a one-case formative OSCE designed to assess communication and interview skills. The encounter was scored by the SP, using a checklist and rating scale form developed at the medical school. The OSCE encounter was also videotaped. Over the course of the academic term, students also spent time in physician preceptor offices as part of a long-term clinical experience curriculum. At the conclusion of the term, preceptors evaluated their students using a checklist and rating scale that assessed student skills along a number of dimensions.

Two types of analyses were conducted in the validation study. For one analysis, the three measures of communication skill: the PVI, the OSCE, and the preceptor evaluation were analyzed for the degree to which the proportion of variance in each overlapped. For the second analysis, the PVI and the OSCE videotape were both scored using the measurement system developed for the PVI. It was hypothesized that using the same measurement system for both the PVI and the OSCE (that is, holding constant the measurement system) would decrease two sources of variability: differences in scorers and differences in scoring systems. Since the OSCE was recorded on videotape, it was available for later analysis.

Canonical correlation was used as an initial rank reduction step to examine the relationships among the measures. Canonical correlation is a technique to answer questions concerning the degree to which a set of variables can be predicted or explained by another set of variables and the degree to which the two sets of variables overlap. [13] Percent redundancy is the statistic used to explain the proportion of variance in each set that is explained by the opposite set. Canonical correlation was used as an omnibus test to identify a small number of variables that could then be examined with Pearson product-moment correlations to explore relationships among the variables. Correlations required for significance equivalent to an alpha of .05, one-tailed, were adjusted for planned comparisons.

Results

Each PVI and each OSCE videotape was rated by two raters. Applying a formula for correction of restriction of range, [14] overall inter-rater agreement correlation was .73. In a comparison between the PVI and the OSCE, 21% of the variance in the OSCE was explained by the PVI and 22% of the variance in the PVI was explained by the OSCE, each as originally scored. This degree of overlap qualifies as substantial. [15] There were no significant correlations between the individual scales in the OSCE as originally scored and the scales of the PVI.

When the measurement system was held constant, by using the same measurement system for both the PVI and the OSCE videotape, the explained variances were 23% and 24%, respectively. Thus, the overlap between the PVI and OSCE was essentially the same regardless of the measurement system. The difference in how the variance was accounted for between the two types of analyses was considerable, however. When the measurement system was held constant for both instruments, significant correlations emerged encompassing measures of empathy (correlation equaled .45), the reciprocal relationship between dominance and friendliness (correlation equaled -.38), and talking about the patient's feelings (correlation equaled .61).

In a comparison between the PVI and the preceptor evaluation, 18% of the variance in the preceptor evaluation was explained by the PVI and 4% of the variance in the PVI was explained by the preceptor evaluation. There were no significant correlations between the preceptor evaluation and the PVI.

Discussion

The PVI was found to overlap substantially with a one-case OSCE as a measure of communication skills. Applying the two types of scoring systems (as originally scored and held constant across measurement systems) resulted in similar degrees of overlap between the PVI and the OSCE, but the contribution of individual variables to the overlap became clearer when the scoring system was held constant. That is, by decreasing sources of error, we can enhance understanding of what is actually being measured. The PVI was found to have little overlap with preceptor evaluations. Each measurement instrument: the PVI, the OSCE, and the preceptor evaluation, is designed to capture a variety of information. Although communication skills are, to varying degrees, a focus of each measure, both OSCEs and preceptor evaluations have broader goals, while the PVI is designed to focus more exclusively on communication skills.

There are two particular limitations relevant to this study. One is the fact that the three measures (the PVI, the OSCE, and the preceptor evaluation) were conducted at different times over the course of the medical school term. However, since significant overlap between the PVI and the OSCE was found, the goal of this study was accomplished. Had measurements been taken at the same time, it is possible that the degree of overlap would have been even greater. Another limitation has to do with the limits of a one-case OSCE and a two-case PVI in terms of their generalizability. Clearly, one would not want to make these few cases the entire basis for judging a medical student's communication skills. These and similar measures are designed to be used to provide ongoing, formative feedback or to be a part of a set of other existing summative measures. The PVI could also serve as a tool for teaching, providing a way for students to practice communication skills.

Conclusions

Assessment of communication skills must take into account the complex multidimensional nature of the skills being assessed. The present research has examined such assessment in the context of the relationship of the PVI, a measure of medical student communication skills, with other measures of such skills in current use at medical schools. The PVI has been found to have overlaps with the other measures of communication skills in current use. There is evidence of convergent and divergent validity, as demonstrated by the areas of overlap and non-overlap. Sources of variability in the assessment of communication skill have been identified, and it has been shown that decreasing sources of error can enhance understanding of what is actually being measured. It has been noted that the other measures of communication skills in common use, OSCEs and preceptor evaluation, focus only in part on communication skills and have broader areas of interest. The PVI can serve either as a useful addition to existing measures, or it can replace existing measures, in some instances and depending on the focus of interest, in part because it controls for a number of these sources of variability.

This research has examined some of the issues related to the quantification of communication skills and has examined the validity of a new instrument, the Patient Video Interview. Partial overlap between the OSCE and the PVI has been demonstrated. While the goals of the OSCE are broad, the current PVI is designed to focus exclusively on communication skills. The PVI overcomes many of the drawbacks to the use of OSCEs and SP-based assessment, including cost, time at individual sites, and standardization across multiple sites or multiple administrations.

References

[1.] Evans, B. J., Stanley, R. O., & Burrows, G. D. "Communication Skills Training and Patients' Satisfaction." Health Communication, Vol. 4, No. 2, 155-170, (1992).

[2.] Lewis, C. C., Pantell, R. H., & Sharp, L. "Increasing Patient Knowledge, Satisfaction, and Involvement: Randomized Trial of a Communication Intervention." Pediatrics, Vol. 88, No. 2, 351-358, (1991).

[3.] Bartlett E. E., Grayson, M., Barker, R., Levine, D. M., Golden, A., & Libber, S. "The Effects of Physician Communications Skills on Patient Satisfaction, Recall, and Adherence." Journal of Chronic Disease, Vol. 37, No. 9/10, 755-764, (1984).

[4.] Kaplan, S. H., Greenfield, S., Gandek, B., Rogers, W. H., & Ware, J. E. "Characteristics of Physicians with Participatory Decision-making Styles." Annals of Internal Medicine, Vol. 124, 497-504, (1996).

[5.] Santana, S. "Step 2 Clinical Skills Exam Set for June 2004." AAMC Reporter, September, 2003. Accessed 03/26/04 at: http://www.aamc.org/newsroom/reporter/sept03/exam.htm

[6.] Jaklevic, MC. "Making the Grade: US to Put Foreign Medical School Graduates' Skills to Test." Modern Healthcare, Vol. 28, No. 10, (1998).

[7.] Colliver, J. A., & Williams, R. G. "Technical Issues: Test Application." Academic Medicine, Vol. 68, No. 6, 454-460, (1993).

[8.] Petrusa, E. R., Blackwell, T. A., Carline, J., Ramsey, P. G., McGaghie, W., Colindres, R., Kowlowitz, V., Mast, T. A., & Soler, N. "A Multi-institutional Trial of an Objective Structured Clinical Examination." Teaching and Learning in Medicine, Vol. 3, No. 2, 86-94,(1991).

[9.] Reed V. A. "Assessing Communication Skills in Medical Education: Development of the Patient Video Interview." (Doctoral dissertation, Dartmouth College, 1997). Dissertation Abstracts International, Vol. 58, No. 8-B, 4121, (1997).

[10.] Cohen-Cole, S. A. The Medical Interview: The Three-function Approach. St. Louis: Mosby, 1991.

[11.] Robinson, J. P., & Shaver, P. R. (Eds.). Measures of Social Psychological Attitudes. Ann Arbor, MI: Institute for Social Research, The University of Michigan, 1974.

[12.] Boon, H., & Stewart, M. "Physician-patient Communication Assessment Instruments: 1986-1996 in Review." Patient Education and Counseling, Vol. 35: 161-176, (1998).

[13.] Thompson, B. Canonical Correlation Analysis: Uses and Interpretation. Beverly Hills, CA: Sage, 1984.

[14.] Hunter, J. E., & Schmidt, F. L. Methods of Meta-analysis: Correcting Error and Bias in research findings. Newbury Park, CA: Sage, 1990.

[15.] McHugo, G. J. "A Multivariate Analysis of Self-selection and Change Due to College Experience." (Doctoral dissertation, Dartmouth College, 1979). Dissertation Abstracts International, Vol. 40, No. 4-B, 1867-1868, (1979).

Reed, Ph.D., is Research Associate Professor of Psychological & Brain Sciences and Jernstedt, Ph.D., is Professor of Psychological & Brain Sciences at Dartmouth College