What alters physicians' decisions to admit to the coronary care unit?
METHODS. A prospective trial of the decision-support intervention was performed on the Family Practice service at a 1 00-bed rural hospital. Cards with probability charts from the acute ischemic Heart Disease Predictive Instrument (HDPI) were distributed to residents on the service and withdrawn on alternate weeks.
Residents were encouraged to consult the probability charts when making CCU placement decisions. The study decision was between placement in the CCU and in a monitored nursing bed. Analyses included ail patients admitted during the intervention trial year for suspected acute cardiac ischemia (n=89), plus patients admitted in two pretrial periods (n=108 and 50) and one posttrial period (n=45).
RESULTS. In the intervention trial, HDPI use did not affect CCU utilization (odds ratio 1.046, P [is greater than] .5). However, following the description of the instrument at a departmental clinical conference, CCU use markedly declined at least 6 months before the intervention trial (odds ratio 0.165, P [is less than] .001). Simply in learning about the instrument, residents achieved sensitivity and specificity equal to the instrument's optimum, whether or not they actually used it.
CONCLUSIONS. Physicians introduced to a decision-support tool achieved optimal CCU utilization without actually performing probability estimations. This may have resulted from improved focus on relevant clinical factors identified by the tool. Teaching simple decision-making strategies might effectively reduce unnecessary CCU utilization.
KEY WORDS. Medical decision making; chest pain; physicians' practice patterns; coronary care units. (J Fam Pract 1997, 45:219-226)
Decision-support tools to improve the appropriateness of the emergency department disposition of cases of suspected acute cardiac ischemia (myocardial infarction or unstable angina) have been heavily researched over the last two decades.[1-10] One of the motivations for the research on decision support is that educational interventions have generally not yielded significant lasting changes in physician behavior.[11-13]
Unfortunately, while there is extensive literature documenting the validity of decision-support tools for heart disease, they have fared little better than education in effectively changing clinical practice. In two trials that provided decision-support tools for physicians to use or not as they chose, the tools were found to be ineffective; physicians tended not to use them.[15,16] One recent trial providing probability information without human interaction also failed to change behavior. Two trials have demonstrated physician behavior change[2,18]; these trials failed, however, to adequately exclude bias due to nonspecific Hawthorne and sentinel[20,21] effects inherent in how the interventions were applied.
We report a dramatic change in CCU utilization by a family practice teaching service at a small community hospital. We discovered this change while considering the failure of a trial that aimed to influence physician use of coronary care unit (CCU) services by applying the acute ischemic Heart Disease Predictive Instrument (HDPI). Our findings suggest a potential educational strategy for changing physician decision behavior and emphasize the importance of considering nonspecific effects in interpreting decision-support trials.
Setting and Patients
We studied patients with suspected acute ischemic heart disease (AIHD) admitted to the inpatient family Practice (FP) service at a 100-bed community hospital. Located in a town of 4000 population, the hospital serves a surrounding community of about 20,000. The population is 98% white, and 68% of local residents are blue-collar workers. For patients with suspected AIHD, an emergency physician (or occasionally an FP outpatient-clinic physician) initially decides on hospitalization; the senior resident on service, with the approval of the attending physician, then decides whether the patient should be placed in the CCU or in a regular nursing bed with ECG telemetry. No administrative incentives, sanctions, or other activities aimed at reducing CCU utilization were initiated at this hospital during any of the study periods.
A retrospective review of all AIHD admissions to the FP service between January 1984 and September 1985 demonstrated very high CCU utilization. In November of 1987, the lead author presented his findings along with a description of the HDPI at a departmental conference. The HDPI is a logistic formula for calculating the probability that a patient has acute ischemic heart disease. It generates a probability score from seven historical and ECG findings, scored dichotomously as present or absent. Subsequently, the appropriateness of CCU utilization was often questioned at department morbidity and mortality conferences.
The week before July 1, 1988, the lead author sent to all FP residents a memorandum explaining the study, presenting the literature in support of the HDPI, and outlining the dimensions of the problem of inappropriate CCU utilization. Beginning July 1 and continuing for 1 year, we used an ABAB reversal design: pocket-sized plastic-laminated cards bearing tables the HDPI's probabilities (Figures 1 and 2) were alternately distributed and withdrawn weekly. As residents rotated through the service and the call schedule, all were exposed to the cards for 2 of their 4 weeks on service, and on one half of their call nights.
The intervention was carried out between July 1, 1988, and June 30, 1989. The medical records of all FP patients aged 35 and older admitted to the hospital during this period were examined, and those admitted for suspected AIHD were collected. Subsequently, we also identically abstracted records for the 6 months preceding and the 6 months following the intervention. A graduate research assistant abstracted each record for demographic information, admission and discharge diagnoses, CCU utilization, peak creatine kinase (CK) level and MB fraction, complications (sustained ventricular arrhythmias, high-grade block, congestive failure, and reinfarction), and HDPI score. All residents who admitted patients during the intervention trial also took part in an unstructured interview designed to assess the instrument's adoptability potential.
In reporting our results, we refer to four periods in chronological order of admission dates: period 1 comprises the original retrospective data; period 2 encompasses admissions during the 6 months preceding the intervention; period 3 is made up of admissions during the intervention; and period 4 constitutes admissions during the 6 months following the intervention.
Univariate statistical comparisons were performed, using one-way analysis of variance for interval data and chi-square testing for categorical data. Likelihood-ratio chi-square tests were used when 2x2 comparisons were made. The Kolmogorov-Smirnov statistic was used to compare distributions. Logistic regression was used for all multivariable analyses.
Considering only intervention trial (period 3) patients, we tested the intervention study hypothesis by determining whether the reversal phase (the week using the HDPI, as compared with the week not using the HDPI) was a significant predictor of CCU placement in a logistic regression model. Other independent variables in the model were age, patient sex, physician sex, and HDPI score.
The results of the intervention trial suggested that CCU utilization behavior was much different from the behavior previously experienced at this hospital. To elucidate this, we analyzed CCU utilization in a logistic regression model as a function of time, using as independent variables patient age, sex, HDPI score, and dummy variables for periods 1 through 4. We also plotted the sensitivity and specificity of residents' admission decisions against the HDPI's receiver operating characteristic (ROC) curve. The ROC curve graphically displays the tradeoff between sensitivity and specificity when different cutoff points (in this case, HDPI score) are used to denote a positive test. For the resident placement in the CCU was considered a positive test. We designated the occurrence of myocardial infarction (defined as an elevation of CK above 150 IU/L with MB fraction more than 5% of total CK) as true disease. The HDPI predicts acute ischemic heart disease, not just myocardial infarction (MI); however, for consideration of the need for CCU admission, MI serves as a reasonable proxy.
The numbers and characteristics of patients admitted during each of the four periods are displayed in Table 1. Patients did not differ significantly over the periods by HDPI score (F(3) = 1.90, P [is greater than] .2; Kolmogorov-Smirnov D for period 1 compared with periods 2 through 4 = 0.125, P [is greater than] .2), sex [Chi.sup.2](3) = 5.92, P [is greater than] .1), or occurrence of MI [Chi.sup.2](3) = 1.50, P [is greater than] .6). The periods did differ by age (F(3) = 3.61, P=.014). Subsequent analyses used multivariate logistic regression models to control for the age difference between periods.
Logistic Regression Results in Analysis of CCU Utilization Variable Weight Odds Ratio (95% CI) Age 0.012 1.128(*) (0.923-1.38) Male sex 0.81 2.25 (1.26-4.02) HDPI score 2.66 1.95(dagger) (1.39-2.73) Period 2[double dagger] -1.80 0.165(dagger) (0.069-0.398) Period 3$ -1.93 0.145 (0.066-0.316) Period 4$ -1.74 0.175 (0.069-0.443)
(*) Odds ratio for one 10-year increment in age. (dagger) Odds ratio for change in HDPI score of 0.25 (eg, 0.50 vs 0.25). (double dagger) Dummy variables representing the four periods, with period 1 as the reference group.
The 48 patients admitted to the CCU during the intervention trial used a total of 112 days of CCU care. Fifteen (31%) of these patients sustained an MI; 13 of the 15 had been placed in the CCU at admission. Only 4 intervention trial patients suffered complications requiring CCU services; 2 of these patients died. Three of these patients had been placed in the CCU at admission; the fourth (one of the deaths) had requested do-not-resuscitate status.
The hypothesis that use of the HDPI would reduce use of the CCU was not supported: 17 of 30 (57%) patients admitted during weeks using the HDPI and 31 of 59 (53%) admitted during weeks not using it were placed in the CCU. Logistic regression on data from the intervention trial admissions (period 3) disclosed no difference in CCU placement during weeks when residents were using the HDPI compared with those when they were not (odds ratio 1.046, 95% CI 0.36 to 3.0, P [is greater than] .5). A 1000-trial bootstrap(24) procedure was used to estimate that the probability of the observed no-difference result is only 24% if the HDPI produced a 10% reduction in admission rate from a base rate of 60%.
The finding that only 54% of all patients were placed in the CCU during the intervention period, in contrast to the historical pattern of 90% (compare periods 1 and 3 in Table 1), led us to examine data from the other periods. CCU utilization did differ according to period ([Chi.sup.2](3) = 35.8, P [is less than] .001). A second logistic regression using three dummy variables to represent the periods confirmed this impression (Table 2). Controlling for age, sex, and HDPI score, period 2 through 4 patients were substantially less likely to be admitted to the CCU than period 1 patients. The adjusted odds ratios for the three variables representing periods 2 through 4, compared with baseline period 1 patients, ranged from 0.145 to 0.175 (Table 2).
[TABULAR DATA NOT REPRODUCIBLE IN ASCII]
The sharp change in admitting practices between period 1 and the subsequent periods is illustrated graphically in Figure 3. Figure 3 is a receiver operating characteristic (ROC) curve for the HDPI that is also marked with point estimates of the performance of the resident physicians. For both the HDPI and the residents, the diagnosis of MI is used as the outcome measure. For the HDPI, various cut-off levels determined a positive test; for the residents, the decision to admit to the CCU is considered a positive test. Between period 1 and the remainder of the study, the residents increased the specificity of their placement decisions without losing sensitivity.
Using the HDPI as a decision-support tool, we attempted to improve CCU utilization from historically high levels, only to find that utilization had already changed. The change occurred after the resident physicians learned about the HDPI but before they began using it to actually calculate probabilities. The change persisted for months after the intervention was withdrawn.
Trials of decision-support tools typically document a high percentage of incorrect decisions made by physicians, and either (1) improve those decisions, or (2) fail to do so as a result of physician nonuse of the tool. Our failure and the recently reported unsuccessful trial conducted by Lee and associates are intriguingly different from prior results. In these two studies, the intervention failed to improve decision-making at least in part because the historically high percentage of incorrect decisions had disappeared. In our trial, for the 6 months before, the year during, and the 6 months after our intervention trial, residents made decisions at or near the HDPI's optimum Figure 3).
How Did They Do That?
The most interesting findings in research are the unexpected ones, and this outcome was most unexpected. The central surprise is that the residents demonstrated that they could achieve sensitivity and specificity equal to that of a sophisticated regression-based decision-support tool, without actually calculating probabilities. How did they do it?
Let us first dispose of one potential explanation, that the failure of the HDPI to improve decision-making during our trial was an experimental design problem. One might object that the weekly changeover between intervention and control phases allowed "contamination" of the control phase by the intervention phase. This objection fails for two reasons. First, the marked change in CCU admission practices predated the reversal-design intervention (period 3) and persisted after it. Second, the persistence of an effect on weeks when probability calculation was unavailable would demonstrate not "contamination" but the actual calculation of probabilities was unnecessary: that some effect of the HDPI other than its purported decision-support mechanism affected decision-making.
Possible causes for the change include Hawthorne and sentinel effects, a secular trend, and the introduction of the HDPI serving as an unintended and unusual educational intervention. A Hawthorne effect (improvement in performance as a result of increased attention) is unlikely, as the change appeared before period 2 and persisted through period 4, whereas only period 3 contained the direct attention necessary to produce the Hawthorne effect. A sentinel effect (improved performance due to awareness that performance will be reviewed) seems similarly unlikely. After the presentation of the findings of the retrospective review (period 1 data) prior to period 2, low-probability "rule-outs" were often commented on at departmental morbidity and mortality conferences; however, there was no real review of CCU admissions at these conferences or elsewhere, and no consequences attendant upon CCU placements of doubtful necessity. A sentinel effect cannot be completely excluded, however, without a sham-intervention group, which studies of decision support generally lack.
Next let us consider a secular trend. In favor of this explanation are the time between periods 1 and 2 and the turnover of the residents during the interval. Against the secular trend hypothesis are three observations. First, the distribution of HDPI scores did not change across the periods, so the screening of patients in the emergency department was constant. Second, there were no policy changes related to CCU utilization on the part of the department, the residency program, or the hospital during this time. The hospital continued to encourage CCU placement of any patient with chest pain, no matter how low the probability of MI, throughout. Finally, private physicians in this hospital had CCU placement rates similar those of period 1 through the entire time covered in our data, although the behavior of the other hospital physicians may not be completely comparable to that of our residents.
An Unintentional Educational Intervention?
The possibility that our introduction of the HDPI acted as an educational intervention is suggested by recent work in the psychology of judgment and decision-making. Models of "bounded rationality" recognize that human decision-makers have limited ability to attend to multiple cues and that they process information sequentially rather than integratively. Individuals use simple cognitive strategies collectively referred to as "probabilistic mental models," or PMMs.(25) These strategies employ sequential evaluation of small numbers of cues, usually fewer than 10, and such remarkably simple cognitive strategies as tallying, counting only the positive cues, with no attention to negatives.
A surprising recent finding is that under conditions of uncertainty or limited information, such simple heuristics can lead to decisions equal or superior in accuracy to those achieved by calculation of regression-based models or other sophisticated integrative strategies.(26) The performance of PMMs depends little on the actual strategy but strongly on connect choice of cues. In evaluating suspected AIHD patients, we have found that often physicians pay much attention to "pseudodiagnostic" cues, which impairs their diagnostic accuracy. The seven factors that comprise the HDPI are those that were most strongly predictive of ischemia among"(57) variables considered during its development. We believe that exposure to the HDPI changed our residents' admission patterns by teaching them to attend to cues of genuine predictive utility rather than to pseudodiagnostic information.
Could our introduction of the HDPI have been sufficient to bring about this major behavioral change? Although much of the literature on changing physician behaviors has been disappointing, individual or small group sessions to influence drug-prescribing behavior have had significant impact in some settings.(28) Soumerai and Avorn(29) identified 11 important components of successful drug detailing and academic counter-detailing: (1) defining specific problems and objectives; (2) identifying physician motivations for use of a product; (3) establishing credibility; (4) targeting high-potential physicians; (5) involving opinion leaders; (6) two-sided communication; (7) promoting active learner involvement; (8) repetition and reinforcement; (9) use of brief graphic print materials; (10) offering practical alternatives; and (11) selection and training. Introducing the HDPI at a departmental conference and mentioning the appropriateness of CCU admissions in subsequent conferences arguably met many of these criteria (specifically 1, 3 through 8, and 10). Residents had been very high utilizers of CCU services, and a workable alternative (telemetry monitoring outside the CCU) was available. Department conferences were small and relatively interactive in the late 1980s, and the small number of faculty had high credibility with residents. The HDPI's introduction and the conferences appear to have "detailed" the cues of genuine predictive utility.
The unstructured interviews with the FP residents after completion of their time on service lend support to this belief. While the residents could not recall actual probability scores from the chart of predictive instrument probabilities (Figures 1 and 2) they could accurately recall the factors on which the probabilities were based, even several months following completion of the project.
Could a simple cognitive strategy such as a PMM, used by a resident with the correct cues, provide such impressive results? In addition to the residents' performance, Figure 3 is marked with points for the performance of two examples of PMMs: simple tallying, and "take the best." The simple tally is positive if more than two of the HDPI factors are present, negative if not. "Take the best" is positive if the patient has: (1) ST segment changes, or (2) a chief complaint of chest pain plus any one other factor. From the ROC curve, it is apparent that both these simple PMMs performed as well as the HDPI and as well as the residents. More important, both simple PMMs provide examples of how a physician could make decisions equal in accuracy to the HDPI after simply seeing it, ie, without actually calculating probabilities.
These findings are potentially very useful to the primary care physician and to the primary care teacher. The medical decision-making literature contains the unspoken presumption that the human decision process is flawed,(30) and hence must be supplemented by or replaced with decision-analytic or regression-derived decision-support models. Our unexpected findings suggest, on the contrary, that the physician's decision process may perform as well as the best available logistic regression model in at least some situations. The key to such performance is selection of the correct cues. A validated decision-support tool can identify those cues, and a simple intervention can communicate them effectively.
Unexpected findings are seldom unequivocally interpretable, as the experiments in which they were discovered were not designed with them in mind. Our results are no exception. Although we can show the educational hypothesis to be plausible, we cannot exclude all other possible explanations. Our data make it more likely than other possibilities. This study was performed in a small community hospital, the study problem is well defined and circumscribed, and there is a good decision-support tool correctly identifying the important clinical data. To what extent our findings would hold in other settings remains unclear.
In a setting where a pattern of excessive use existed previously, we found that optimal use of the CCU was achieved following education about, but without the actual use of, a decision-support tool. Explicit probability calculation was not necessary in order to change decision-making.
Existing decision-making research tends to consider typical clinical decision-making inherently defective, and seeks to replace or reform it. These results suggest the possibility of developing strategies, based on the latest judgment and decision-making theory, that could build on the strengths of, rather than seek to replace, clinicians' reasoning. Such strategies could be taught quickly and at low cost. How broadly this technique might be applied remains to be determined.
[1.] Pozen M, D'Agostino R, Mitchell J. The usefulness of a predictive instrument to reduce unappropriate admissions to the coronary care unit. Ann Intern Med 1980; 92:23842.
[2.] Pozen M, D'Agostino R, Selker H. A predictive instrument to improve coronary-care-unit admission practices in acute ischemic heart disease: a prospective multicenter clinical trial. N Engl J Med 1984; 310:1273-8.
[3.] Brush JJ, Brand D, Acampora D. Use of the initial electrocardiogram a predict in-hospital complications of acute myocardial infarction. N Engl J Med 1985; 312:1137-41.
[4.] Tierney W, Roth B, Psaty B. Predictors of myocardial infarction in emergency room patients. Crit Care Med 1985; 13:526-31.
[5.] Goldman L, Weinberg M, Weisberg M. A computer-derived protocol to aid in the diagnosis of emergency room patients with acute chest pain. N Engl J Med 1982; 307:588-96.
[6.] Lee T, Cook E, Weisenberg M. Acute chest pain in the emergency ward: identification and evaluation of low risk patients. Arch Intern Med 1985; 145:65-9.
[7.] Diamond G, Forrester J. Analysis of probability as an aid in the clinical diagnosis of coronary-artery disease. N Engl J Med 1979; 300:1350-8.
[8.] Rouan G, Lee T, Cook E, Brand D, Weisberg M, Goldman L. Clinical characteristics and outcome of myocardial infarction in patients with initially normal or nonspecific electrocardiograms (a report from the Multicenter Chest Pain Study). Am J Cardiol 1989; 64:1087-92.
[9.] Green L, Smith M. Evaluation of two acute cardiac ischemia decision-support tools in a rural family practice. J Fam Pract 1988; 26:627-32.
[10.] Davison G, Suchman Al, Goldstein BJ. Reducing unnecessary coronary care unit admissions: a comparison of three decision aids. J Gen Intern Med 1990; 5:474-9.
[11.] Schroeder SA. Strategies for reducing medical costs by changing physicians' behavior efficacy and impact on quality of care. Int J Technol Assess Health Care 1987; 3:39-50.
[12.] Eisenberg JAL Physician utilization. The state of research about physicians' practice patterns. Med Care 1985; 23:461-83.
[13.] Axt-Adam P, van der Wouden JC, van der Does E. Influencing behavior of physician ordering laboratory tests: a literature study. Med Care 1993; 31:784-94.
[14.] Lee TH. Evaluating decision aids: the next painful step. J Gen Intern Med 1990; 5:528-9.
[15.] Corey GA, Merenstein JH. Applying the acute ischemic heart disease predictive instrument. J Fam Pract 1987; 25:127-33.
[16.] Pearson SD, Goldman L, Garcia TB, Cook EF, Lee TH. Physician response to a prediction rule for the triage of emergency department patients with chest pain. J Gen Intern Med 1994; 9:241-7.
[17.] Lee TH, Pearson SD, Johnson PA, et al. Failure of information as an intervention to modify clinical management. A time-series trial in patients with acute chest pain. Ann Intern Med 1995; 122:434-7.
[18.] Sarasin FP, Reymond JM, Griffith JL, et al. Impact of the acute cardiac ischemia time-insensitive predictive instrument (ACI-TIPI) on the speed of triage decision making for emergency department patients presenting with chest pain: a controlled clinical trial. J Gen Entern Med 1994; 9:187-94.
[19.] Cook TD, Campbell DT. Quasi-experimentation: design and issues for field settings. Boston, Mass: Houghton-Mifflin, 1979.
[20.] McCarthy EG, Fuikel MI, Ruchlin HS. Second opinion elective surgery. Boston, Mass: Auburn House Publishing, 1981.
[21.] Donabedian. A. The methods and findings of quality assessment and monitoring: an illustrated analysis. Ann Arbor, Mich: Health Administration on Press, 1985.
[22.] Selker H. Sorting out chest pain: identifying acute cardiac ischemia in the emergency room setting. Emerg Dec 1985 Jun:8-17.
[23.] Becker M. Factors affecting diffusion of innovations among health professionals. Am J Public Health 1970; 60:2294-304.
[24.] Simon JL. Resampling: the new statistics. Arlington, Va: Resampling Stats, 1992.
[25.] Gigerenzer G, Hoffrage U, Kleinbolting H. Probabilistic mental models: a Brunswikian theory of confidence. Psych Rev 1991; 98:506-28.
[26.] Gigerenzer G. The psychology of good judgment: frequency formats and simple algoriths. Med Decis Making 1996; 16:273-80.
[27.] Green LA, Yates JF. Influence of pseudodiagnostic information on the evaluation of ischemic heart disease. Ann Emerg Med 1995; 25:451-7.
[28.] Soumerai SB, McLaughlin TJ, Avom J. Improving drug prescribing in primary care. A critical analysis of the experimental literature. Milbank Q 1989; 67:268-317.
[29.] Soumerai SB, Avom J. Principles of educational outreach (`academic detailing') to improve clinical decision making. JAMA 1990; 263:549-56.
[30.] Gigerenzer G. How to make cognitive illusions disappear. beyond "heuristics and biases." Eur Rev Soc Psychol 1991; 2:83-115.
|Printer friendly Cite/link Email Feedback|
|Author:||Green, Lee; Mehr, David R.|
|Publication:||Journal of Family Practice|
|Date:||Sep 1, 1997|
|Previous Article:||Patient preferences for colorectal cancer screening.|
|Next Article:||NSAIDs increase risk of gastrointestinal bleeding in primary care patients with dyspepsia.|