Interrater reliability of auscultation of breath sounds among physical therapists.

[Brooks D, Thomas J. Interrater reliability of auscultation of breath sounds among physical therapists. Phys Ther. 1995; 75:1082-1088.]

Key Words: Auscultation, Physical Therapy, Reliability.

Medical professionals, including physical therapists, rely on auscultation in the day-to-day assessment of patients. The reliability of judgments made using auscultation, however, has not been well documented.

As early as 1969, Forgacs described the stethoscope as "a largely decorative tool in so far as its value in diagnosis of pulmonary disease is concerned."[1(p11]) Since that time, both clinicians and researchers have recognized that several factors may impede the reliability and validity of auscultation. The disparity in nomenclature among health care professionals may result in decreased reliability because different terminology is used to describe abnormal sounds, such as the use of "rales" and "rhonchi" synonymously for "crackles" and "wheezes." Furthermore, the quality of the stethoscope can also have an effect on reliability; scientific and clinical evidence indicates that sound transmission may differ in the different types of stethoscopes.[2] Finally, because a definitive correlation between pathological findings and lung sounds remains to be established, the validity of this measure is unclear.

Several studies have examined the reliability of auscultation.[3-5] Aweida and Kelsey[3] examined the accuracy and reliability of physical therapists in auscultating tape-recorded lung sounds. Analysis of interrater reliability using the kappa statistic showed only a fair degree of agreement among subjects. A follow-up study by Brooks and colleagues[4] showed similar reliability in a group of experienced physical therapists. Pasterkamp and colleagues[5] examined the nomenclature used by different health care professionals during auscultation of tape-recorded lung sounds and the resulting observer variability. Of tie 40 health care professionals who participated in their study, 10 were licensed physical therapists. Their study revealed differences between groups of observers in the terminology used to describe adventitious sounds, and physical therapists were found to be the most consistent with regard to their individually chosen nomenclature.

In all of these studies,[3-5] tape-recorded lung sounds were used. The rationale for the use of tape-recorded lung sounds is that they. provide a stable and repeatable signal. In a patient, repeated,inspiration may alter the underlying breath sounds, rendering it difficult to examine reliability. The disadvantage, however, of using tape-recorded sounds is that they may not be reflective of in vivo sounds.

Thus, we set out to examine in vivo reliability of auscultation using a teaching stethoscope with four sets of ear-pieces that allows four therapists to simultaneously auscultate a patient's chest. The purpose was to determine the effects of two factors, clinical experience in chest physical therapy and an education session, on interrater reliability of auscultation among physical therapists. The education session consisted of the adoption of standardized nomenclature published by the American College of Chest Physicians-American Thoracic Society[6] and education on proper technique of auscultation and interpretation of findings.[7]

Methodology

All physical therapists and patients were asked to give written informed consent prior to participation in the study.

A group of 57 registered physical therapists at our facility were given a questionnaire regarding age, number of years of clinical experience in chest physical therapy, their present area of practice, and any hearing impairment. The potential subjects were stratified by clinical experience into four groups. Sixteen subjects were randomly chosen from the 57 therapists using a random number table. Because the therapists were to use a teaching stethoscope with only four pairs of earpieces, there were 4 therapists in each stratum (ie, level of experience). All physical therapists reported using auscultation of lung sounds in 100% of their chest assessments. The following criteria were developed to delineate clinical experience. Group 1 subjects were senior chest physical therapists with at least 5 years of experience in this area of practice. Group 2 subjects were experienced therapists who had a minimum of 2 years of experience in chest physical therapy and were currently practicing in the area. Group 3 subjects were experienced therapists in other areas who only practiced in chest physical therapy on weekend service. Group 4 subjects were new graduates (less than 1 year since graduation) with at least 1 month of experience in chest physical therapy. Group 4 was included because we believed that the subjects' recent academic education in current terminology and technique may differentiate them from subjects in the other groups. The exclusion criterion for the subjects was having any auditory dysfunction.

Potential patients were screened by the investigators or a research assistant to determine their eligibility to participate in die study. The patients selected had to be currently admitted as inpatients with or without compromised respiratory status. They had to be able to follow instructions and sit for 15 minutes at the side of the bed to allow adequate time for auscultation. The exclusion criteria for the patients were an inability to follow instructions, medical instability, or being mechanically ventilated. The presence of abnormality was determined based on the history; chart review; chest radiograph findings; and physical examination of the chest, including auscultation. All attempts were made to ensure that the patient sample represented the spectrum of lung sounds, including normal and abnormal sounds. Prior to examination by the therapists, the investigators or the research assistant drew a circle using a grease pencil over the area of the chest where the desired abnormal or normal lung sound could be heard. To ensure minimal change in the patient's condition, there was a delay of no more than half an hour between the screening and the assessment by the physical therapists.

Ten patients were evaluated by each group of four physical therapists. The same patients were not necessarily used by all four groups of therapists because a patient's status or schedule did not permit multiple assessment if one of the therapists was familiar with the patient's condition. The patients auscultated by each of the groups represented a similar spectrum of lung sounds, with the same number of normal and abnormal findings. Examination of each patient occurred in a quiet room with the patient sitting at the bedside. Only the four therapists in a group and one of the investigators were present in the room during the examination. Therapists were allowed to auscultate the designated area, the corresponding area on the contralateral side, and above and below these areas. The patient was instructed to take several deep breaths through the mouth.

A teaching stethoscope(*) was modified to allow four therapists to auscultate at the same time. The stethoscope had four pairs of earpieces and one diaphragm/bell and was pretested to ensure proper transmission of sounds. The therapists were asked to choose a pair of binaural earpieces from an assortment of many, to maximize comfort and ensure proper transmission of sound.

The therapists were asked to record their findings privately outside the patient's room. Therapists were allowed to use their own terminology when describing their findings. They were instructed to include a description of the quality of lung sounds heard, the adventitious sounds (eg, crackles and wheezes), and the phase of respiration in which they occurred. The investigators ensured that no discussion of the findings occurred among the therapists.

Within 1 week of the assessment, a 1 1/2-hour education session for all 16 physical therapists was scheduled and included a discussion of die following: (1) the standardized nomenclature published by the American College of Chest Physicians-American Thoracic, Society,[6] Which suggests the use of the terms "crackles" and "wheezes" rather than "rales" and "rhonchi"; (2) a description of the underlying pathology of adventitious sounds; and (3) the relevance of the phase of respiration during which the sounds occurred. A commercially available audiotape of the different breath sounds was used to provide examples of the sounds.[7] Therapists also auscultated each other's chests for examples of normal sounds and bronchial breath sounds (over the trachea). Ample time for questions and discussion was available. Although we did not specifically examine whether one session was satisfactory to refresh the therapists' knowledge of auscultation, they reported being comfortable with the material at the end of this session.

Within 1 week from the education session, the same protocol described earlier was repeated (ie, another 10 patients were assessed by each group of therapists). The patients represented the same spectrum of lung sounds as was present prior to the education session. The therapists were asked to use the standardized nomenclature discussed during the education session when recording their findings.

Data Analysis

The independent variables in this study were the levels of clinical experience and the introduction of the education session. The dependent variable was a measure of reliability as reflected in the kappa statistic ([kappa]), which was used to analyze and interpret all data.[8] The analysis included between-group comparisons (performed both before and after education) to determine the effect of clinical experience. Within-group comparisons (performed both before and after education) were performed to determine the effect of the education session on reliability.

The kappa statistic was used, which implicitly assumes that afl disagreements are equally serious.[9] Because we could not specify the relative seriousness of each type of disagreement, we did not use the weighted kappa statistic. For example, when describing crackles, if the three raters respectively described them as crackles on inspiration versus crackles on expiration versus crackles on inspiration and expiration, the disagreement among the three raters was considered equal. The kappa statistic is scaled to vary from -1 to +1, where a negative value indicates less than chance agreement, 0 is chance agreement, and a positive value indicates greater than chance agreement. The kappa statistic accounts for the number of categories as well as the number of trials repeated (ie, representing the 10 patients in this study).[9] The properties of this experiment for deriving the kappa statistic are described as multinomial (ie, the experiment consisted of several independent trials, with a set number of outcomes for each trial and a set probability for each outcome.10 For comparisons of findings between trials, the chi-square and z statistics are used.[10]

First, we examined the reliability of physical therapists in their ability to distinguish any abnormal lung sound from normal lung sounds. The z statistic, which allows for comparison of two samples, was used to compare kappa values obtained before the education session with those obtained after the education session.[11] Because there were four groups of therapists with different levels of clinical experience, the reliability between groups was examined using the chi-square statistic to determine whether the kappa values-were different.[11] The z statistic could not be used for this comparison as it only allows for two-sample comparisons. If a statistically significant difference was detected, however, then pair-wise comparisons among the groups were performed using the z statistic. The sample size for patients (N=10) was considered adequate based on Soeken and Prescott,[12] who suggested that more than [2c.sup.2] subjects (where c is the number of categories in the assessment tool) are needed for a valid kappa value. in our study, there were two categories: normal lung sounds and abnormal lung sounds.

Second, we examined the preliminary data on the reliability when detecting abnormal lung sounds. The nomenclature used by the subjects was translated into a number sequence for calculation of the kappa statistic. Table 1 shows the coding used for this translation and provides an example of coding. The reliability between groups for a given variable was examined using the chi-square statistic to determine whether the kappa values were different.[11] The effect of the education session on reliability was examined using z values for each variable within a group. A difference was considered statistically significant for a critical probability value of .05. Given the large number of categories, the sample size did not fulfill the criteria of Soeken and Prescott.[12]

Table 1. Categories Used to Code the
Data Recorded by the Physical Therapists
Into a Five-Digit Sequence(a)
Digit       Category                           Code
First       No crackles                        0
            Coarse crackles                    1
            Fine crackles                      2
            Fine and coarse crackles           3
Second      Crackles on inspiration (I)        0
            Crackles on expiration (E)         1
            Crackles on I and E                2
Third       No wheezes                         0
            High-pitched wheezes               1
            Low-pitched wheezes                2
            High- and low-pitched
              wheezes                          3
Fourth      Wheezes on I                       0
            Wheezes on E                       1
            Wheezes on I and E                 2
Fifth       Normal breath sounds               0
            Deceased breath sounds             1
            Absent breath sounds               2
            Bronchial breath sounds            3
(a) For example, a response of "coarse inspiratory
crackles, expiratory low-pitched wheezes,
and deceased breath sounds" would be coded
10211.

Results

Table 2 shows the results from the analysis of the reliability of detecting abnormal from normal sounds (dichotomous categories). Before the education session, group 3 (specialists in other areas of physical therapy) showed the highest levels of reliability, with a kappa value ([kappa]=.84) that was different than kappa values in all other groups (P<.04). Following the education session, group 3 (specialists in other areas of physical therapy) continued demonstrate die best reliability ([kappa]=.99), differing with group 2 (ie, specialists but not seniors in chest physical therapy; P=.01). Although reliability improved in all four groups following the education session, only group 4 (new graduates) reached statistical significance (P=.0001).

[TABULAR DATA 2 OMITTED]

The second analysis examined the reliability in detecting abnormal lung sounds. The results are presented in Tables 3 and 4. [TABULAR DATA 3 & 4 OMITTED]

Before the education session (Tab. 2), reliability was relatively low for all groups and all variables ([kappa] =-.02-.59). There was a difference between the four groups for variables "crackle," "wheeze," "phase of wheeze," and "breath sounds" (P<.05). Pair-wise comparisons among the groups for these four variables were performed. For the variable "crackle," the kappa value for the new graduates was different than kappa values for all other groups (P<.005). For the variables "wheeze" and "phase of wheeze," group 4 (new graduates) demonstrated the lowest reliability. The variable "breath sounds" demonstrated the lowest reliability among other variables. Groups 1 (seniors in chest physical therapy) and 2 (specialists in other areas of physical therapy) showed the highest kappa values for this variable, which were different than kappa values in groups 3 and 4 (P<.015).

After the education session (Tab. 4), there was a difference between the four groups for "phase of crackle," "wheeze," and "phase of wheeze" (P<.015). Pair-wise comparisons among the groups for these four variables were performed. For the variable "phase of crackle," the kappa value for group 4 (new graduates) was different than kappa values for all other groups (P<.005). For the variables "wheeze" and "phase of wheeze," group 4 (new graduates) demonstrated the lowest reliability, having kappa values that were different than kappa values for all other groups (P<.002).

In general, reliability after the education session improved. Table 5 presents the probability values for the comparisons from before to after the education session within each group using the z statistic. The most improvement in reliability was detected for the variable "breath sounds," which showed a change in kappa values for afl four groups. In group 4 (new graduates), there was deterioration in reliability for the variables "wheeze" and "phase of wheeze."

Discussion

This study examined the reliability of judgments made by physical therapists auscultating lung sounds. The effects of both clinical experience and an education session were assessed. Our findings show a positive effect of an education session on reliability but no clear effect of clinical experience.

When assessing the reliability of assessments by physical therapists in detecting the presence of any abnormal lung sound, the reliability of auscultation was fair. Surprisingly, therapists who were specialists in other areas of physical therapy showed the best reliability; however, the higher degree of reliability did not persist. when the identification of the specific abnormality was required. The education session resulted in improvement in the reliability of assessments by new graduates in detecting the presence of any abnormal lung sound.

When assessing the reliability of judgments made by physical therapists in detecting abnormal lung sounds, interrater reliability of auscultation was poor to fair, in general, before the education session. The education session resulted in an overall improvement in reliability. New graduates and seniors in chest physical therapy showed the greatest agreement prior to the education session in detecting specific abnormalities. This result may reflect a current knowledge of terminology within these groups. Physical therapists who were specialists in other fields appeared to benefit the most from the education session on specific abnormalities. This finding may be due to the fact that they rarely assess respiratory status and have not established a method of assessment and specific nomenclature for identifying lung sounds, and were therefore more open to changes. Specialists in chest physical therapy showed the least amount of change following the education session. This finding may have occurred because their technique was more established and therefore more difficult to change. The agreement among observers appeared to depend on the variable considered. This finding is similar to the results of Pasterkamp and colleagues,[5] who reported variability for the characteristic of wheezing.

In our study, we chose to use in vivo sounds rather than tape-recorded lung sounds because the quality of tape-recorded lung sounds is not truly representative of what is heard through a stethoscope in a clinical setting in an ideal reliability study, all therapists would listen to the same lung segment of a patient at the same time. In the clinical setting, even if the therapists listened to the lung sounds consecutively, the sound heard may differ because repeated inspirations may alter the underlying pathology and therefore the lung sound. The presence of upper-airway secretions, for example, may be manifested with low-pitched wheezes; repeated deep breaths result in movement of the secretions and alterations of the sounds heard. For this reason, a teaching stethoscope was used in this study, allowing the therapists to auscultate lung sounds simultaneously.

The finding of low to poor reliability is in agreement with the findings of previous studies on the reliability of auscultation of lung sounds.[3-5] Aweida and Kelsey,[3] who used tape-recorded lung sounds, reported fair reliability with a weighted kappa of .22. A follow-up study by Brooks and colleagues4 demonstrated similar results (weighted kappa value of .20) in a group of experienced physical therapists. Similarly, Pasterkamp and colleagues[5] observed differences between groups of observers in the terminology used to describe adventitious sounds, and they found physical therapists to be the most. consistent with regard to their individually chosen nomenclature. In all of these studies, tape-recorded lung sounds were used, which, as already mentioned, may not be reflective of in vivo sounds.

The poor reliability that we observed for the variable "breath sounds" (categories: normal, bronchial, decreased, absent) is in agreement with previous studies on in vivo auscultation to determine proper intubation. When confirming proper intubation, auscultation is performed to detect whether breath sounds are normal, decreased, or absent. Andersen and Hald[13] reported that auscultation of the lungs to confirm proper intubation of the trachea was inaccurate in 15% of the cases. Similarly, Alliaume and colleagues[14] reported that auscultation was an unreliable method in confirming the position of double-lumen endobronchial tubes in patients undergoing aortic or lung surgery. The inaccuracies detected by Andersen and Hald and Alliaume et al are in agreement with the poor reliability observed in our study for detecting and categorizing breath sounds.

Only one study has examined the effect of education and training on reliability of auscultation. Pasterkamp and colleagues[5] investigated the effect of additional specific instructions on standard nomenclature on intraobserver reliability and found that there was little difference in agreement between spontaneously chosen terms and the standardized terminology. Significant variability, however, was found when the subjects were asked to use Forgacs' terminology[15] for characterization of wheezing as monophonic, polyphonic, high pitched, and low pitched. In our study, reliability was also dependent on the lung sound heard.

Our study did not show a direct relationship between the degree of clinical experience and reliability of auscultation. This finding is contrary to the results of Godfrey and colleagues,[16] who examined the observer variation of physical signs of airway obstruction among physicians and suggested that clinical experience may have an effect on reliability. Our results are in agreement with those of Brooks and colleagues,[4] who compared interrater reliability among specialized and nonspecialized physical therapists in the auscultation of tape-recorded lung sounds and found no differences between the two groups. Experience in chest physical therapy in our study was determined based on die number of years of experience in the field and the area of present practice. We did not account for the educational background of the therapists, recent courses that they may have attended, and the type of patients they were primarily treating. All these factors may also influence reliability.

There are several limitations of this study that must be considered when interpreting the results. Although all attempts were made to ensure that the patient population represented the spectrum of lung sounds, the range of lung sounds could have varied among groups and before and after the education session. In addition, although the sample size of the patients (N=10) was adequate for the analysis of reliability for the dichotomous categories (normal versus abnormal lung sounds), this sample size was too small for the analysis of specific types of abnormal sounds. The effect of small sample size may act to underestimate the reliability estimate. Thus, the kappa value may not have been a good reflection of the reliability for the specific categories. Nevertheless, the large number of patients required to test the reliability in detecting a specific abnormality would require a large commitment of the therapists' time and would be prohibitive in our clinical setting. Furthermore, multiple comparisons were made, which increases the likelihood of observing a positive result when one is not present. For this reason, the actual probability values were provided to allow the reader to assess the strength of significance. Finally, physical therapists were all affiliated with the same institution. Therefore, caution must be exercised when applying these results to the general physical therapist population.

Conclusions

With the emphasis on evaluating the effectiveness of physical therapy, the reliability of assessment tools and measures must be critically examined. Auscultation is possibly one of the most commonly used assessment tools in chest physical therapy. This study has shown fair interrater reliability among physical therapists when auscultating lung sounds. An education session resulted in improvement in reliability. The finding of poor reliability in some of the subgroups in this study suggest the importance of using other tools, in conjunction with auscultation, when assessing @the respiratory status of the patient. Our results also suggest the role of education sessions to improve reliability. Further research is needed to investigate the measurement qualities of auscultation, specifically interrater reliability in the general population of physical therapists, intrarater reliability among physical therapists, and the validity of this assessment technique.

Acknowledgments

We acknowledge the assistance of Shannon Tooke, who was then a fourth-year physical therapy student at the University of Toronto, in data collection. And Dr Paul Corey for statistical support. We also thank all the physical therapists who participated in this study for their support.

(*) Model 2138, 3M General Offices, Medical-Surgical Division, St Paul, MN 55144.

References

[1] Forgacs P. Lung sounds. Br J Dis Chest. 1969;63:1-12. [2] Ertel PY, Lawrence M, Brown RK, Stern AM. Stethoscope acoustics, I: the doctor and his stethoscope, II: the transmission and filtration patterns. Circulation. 1966;34:889-909. [3] Aweida D, Kelsey CJ. Accuracy and reliability of physical therapists in auscultating tape-recorded lung sounds. Physiotherapy Canada. 1990;42,.279-282. [4] Brooks D, Wilson L, Kelsey CK. Accuracy and reliability of "specialized" physical therapists in auscultating tape-recorded lung sounds. Physiotheraphy Canada. 1993;45:21-24. [5] Pasterkamp H, Montgomery M, Wiebicke W. Nomenclature used by health care professionals to describe sounds in asthma. Chest. 1987;92:346-352. [6] ACCP-ATS joint Committee on Pulmonary Nomenclature, Pulmonary Terms and Symbols. Chest. 1975;67:5-10. [7] Lehrer S. Understanding Lung Sounds [book and videotape]. Toronto, Ontario, Canada: WB Saunders Co; 1985. [8] Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-174. [9] Fleiss JL, Cohen J; The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement. 1973; 33:613-619. [10] McClave JT, Dietrich FH. Statistics. 3rd ed. San Francisco, Calif: Dellen Publishing Co; 1985. [11] Rosner B. Fundamentals of Biostatistics. Boston, Mass: PWS-Kent Publishing Co; 1990. [12] Soeken KL, Prescott PA. Issues in the use of kappa to estimate reliability. Med Care. 1986;24:733-741. [13] Andersen KH, Hald A. Assessing the position of the tracheal tube: the reliability of different methods. Anaesthesia. 1989;44:984-985. [14] Alliaume B, Coddens J, Deloof T. Reliability of auscultation in positioning of doublelumen endobronchial tubes. Can J Anaesth. 1992;39:687-690. [15] Forgacs P. Lung Sounds. London, England: Bailliere Tindall; 1978:44-53. [16] Godfrey S, Edwards RTH, Campbell EJM, et al. Repeatability of the physical signs in airway obstruction. Thorax. 1969;24:4-9.

D Brooks, MSc, BSc(PT), is Lecturer and Course Coordinator, Department of Physical Therapy, University of Toronto, 256 McCaul St, Toronto, Ontario, Canada M5T 1W5 ([e-mail]). Address all correspondence to Ms Brooks.

J Thomas, MSc, BSc(PT], was Research Coordinator, Physiotherapy Department, The Toronto Hospital, Toronto, Ontario, Canada, at the time of completion of this study.

This study was approved by The Toronto Hospital Ethics Review Board.

Funding for this study was provided by the Ontario Lung Association through an Ontario Respiratory Care Society research grant.

This article was submitted November 28, 1994, and was accepted August 4, 1995, to allow adequate time for auscultation.