An overview for medical directors.

An Overview for Medical Directors--Part One

Outcome monitoring in the delivery of health care services addresses the critical concerns of effectiveness, appropriateness, and efficiency. Effectiveness is the determination of whether measurable outcomes have been achieved. Thus, all outcome monitoring systems must, in some measure, assess effectiveness. Appropriateness involves weighing the risks and benefits of medical intervention. Because care must be effective in order to be appropriate, outcome monitoring can assist in determining appropriateness. Similarly, effectiveness is a requisite for efficiency. Low cost care is of little value unless the desired outcome is achieved. So the monitoring of outcomes also contributes to assessment of resource utilization.

The outcome of a medical intervention may not ultimately be determined by the skill of the physician or by the teamwork of the hospital staff, but rather by factors such as the patient's age, chronic conditions, secondary illnesses, etc. These factors affect the "material input" that limits or enhances the health care provider's probability of success. Research studies control material input by strictly defining the characteristics of patients to be studied and excluding all others and by randomizing patients into treatment and control groups. Because providers of hospital care have little control over patient input, adjustments must be made in outcome monitoring to compensate for risk factors that are present in only some patients. Ideally, the relative risks of a variety of outcomes would be considered. Thus, a patient's probability of dying might be described as low, while his or her probability of functional recovery might be moderate. The tendency has been to develop methods that describe risk in terms of a single index known as "severity of illness." While using the terms "risk" and "severity" interchangeably in this discussion, we will always specify which outcome or outcomes served as the original basis for each severity index.

Like seeds sprouting in fertile soil, a variety of "severity systems" has emerged to aid and abet (or alternately to confuse and complicate) the medical director's task of risk adjustment for outcome monitoring. From analytic methods requiring only a hand calculator or personal computer to complex automated systems employing sophisticated computing capabilities, all are schemes for factoring out the differences among patients that affect the outcome of their care but that are beyond providers' control.

Like seeds, also, severity systems sprang from different origins and are best used in harmony with their developmental design. For example, systems designed to predict costs will probably do a much better job in connection with resource utilization than they will in predicting mortality. No one system is clearly superior to the others for all uses.

In evaluating severity systems, it is helpful to consider how and why each was developed; what data are required; how each system functions; what is known about its validity, its relative costs (including staff time required for data collection and computer functions), and its limitations; and the importance of comparative findings derived from each system's use. Many other factors, such as the selection and weighting of data elements, are important determinants of a system's performance, but differences in these areas often are highly technical and tend to confuse, rather than enlighten, prospective users.

HCFA's Severity

Adjustment Techniques

The most widely applied severity adjustment system is used by the Health Care Financing Administration to produce its annual Medicare mortality release. Because it has nearly universal application and is a transparent system, being in the public domain, it can serve as a useful standard in comparing other methods and commercial systems, as well as in illustrating the issues that must be considered in selecting a severity system.

* Purpose and Development. Using clinical judgment and actual Medicare data, the HCFA methodology was designed to adjust for a Medicare recipient's risk of dying within 30 days of admission to a hospital.

* Data Collection. The method utilizes administrative data that are almost universally collected, are easily obtained, and are available for facilities other than one's own.

* Technique. The statistical technique employed is logistic regression.

* Experience. Data have been published for all hospitals (more than 5,500 nationwide) that provided acute care to Medicare patients in 1986 and 1987. The database includes more than 12 million patient episodes.

* Cost. The methodology is available to anyone who chooses to read the brief technical appendix printed at the conclusion of Medicare Hospital Mortality Information. [1]

* Validity. Face validity involves the determination of whether the methodology appears, on the surface, to be reasonable. For example, in the case kf real estate, face validity would be the assessment of the house's appearance. Construct validity is the assessment of the methodology's soundness and the logic of its structure. For example, in real estate, the determination of construct validity would involve examination of the house's plumbing, wiring, foundation, heating systems, and perhaps the architectural drawings.

The HCFA methodology has face validity from two standpoints. First, the patient characteristics considered to alter risk are diagnosis, previous hospital admission, age, sex, and the presence of chronic disease. All these factors are known to affect a patient's chance of dying and are used in the HCFA system. Second, when our firm performed a comparative analysis using the HCFA data, we found that the list of 35 hospitals with the lowest mortality rates is dominated by facilities with very high reputations: Massachusetts General Hospital, Boston, Mass.; Mayo Clinic hospitals, Rochester, Minn.; M.D. Anderson Hospital, Houston, Tex.; University of California Health Sciences Center, Los Angeles; Pres byterian Hospital, New York; Barnes Hospital, St. Louis, Mo.; Emory University Hospital, Atlanta, Ga.; and other prominent institutions.

Construct validity has been demonstrated, first, by the fact that risk adjustment was derived specifically for the outcomes being monitored, using actual Medicare data combined with expert clinical judgment. Additional support for model validity is the excellent year-to-year consistency of the data. Fewer than one percent of 5,577 hospitals showed statistically significant change from one year to the next, and preliminary comparisons with 1985 data showed similar consistency. Fifteen percent of variation among hospitals can be expected from chance alone. Of the remaining variation, 59 percent is explained by HCFA's risk adjustment factors, and the remaining 41 percent is due to a combination of quality differences, coding errors, and inadequate risk adjustment. One can only guess how much is due to each, but detailed studies are under way at HCFA to shed light on this issue. Careful examination of individual facilities has, in many cases, yielded probable explanations for variations.

Finally, it is important to note that the 35 hospitals with the lowest mortality demonstrated wide variation in the severity of illness of their patients. St. Mary's, one of the Mayo Clinic hospitals, had a patient population with an underlying risk only 73 percent of the national average, while M.D. Anderson Hospital treated patients with a risk 86 percent higher than average.

* Importance of Findings. Once the methodology is applied, how useful is the information it yields? Outcome models may have excellent associated risk adjustment, but make little contribution to quality management because kf related insensitivity of the monitor. We tested 30-day Medicare mortality, using a method based on a binomial approximation of variance under null hypotheses that are described elsewhere. [2] We discovered that hospitals could be graded on a scale of one (exceptionally low mortality) to eight (exceptionally high mortality), with 10 percent of the nation's hospitals significantly above average in performance, 77 percent average or near average, and 13 percent below average when data for two years (1986 and 1987) were combined.

* Limitations. Only a single outcome--mortality within 30 days of hospital admission--is considered by the HCFA methodology. Because only administrative ata are utilized, more subtle clinical and physiologic information is not available. Because Medicare patients are primarily elderly, pediatric and obstetrical cases are not represented. These limitations must be weighed against the low cost of the methodology and the advantages of its extensive application.

Severity Adjustment Using

Administrative Data

Other methods of adjusting for patients' differences that, like HCFA's, rely only on administrative data are Diagnosis-Related Groups, Patient Management Categories, Disease Staging, and the PRAGmatic system.

Diagnosis-Related Groups (DRGs)

* Purpose and Development. DRGs were designed to group patients according to their degree of resource consumption. They were developed using iterative techniques that combine clinical knowledge and evaluation of empirical data. Experience has led to further refinements in the form of additional categories.

* Data Collection. Because DRG codes are part of the standard uniform billing discharge summary, no additional data collection is required.

* Technique. Rules govern the classification of each patient into one of approximately 500 groups, either by hand kr by the use of a software program. Each group is intended to be homogeneous in resource consumption.

* Experience. Widely used in hospital accounting, finance, and planning, DRGs are familiar both to administrators and to clinicians as the basis for prospective pricing for Medicare patients over most of the past decade. Because they are universal and in the public domain, DRGs have been incorporated into other risk adjustment systems whose developers were seeking some connection with patients' diagnoses.

* Cost. Marginal cost is minimal, because data collection and classification are already necessary in current operations.

* Validity. Health Systems International (New Haven, Conn.) reviews the grouper and weights annually and modifies them on the basis of cost data gathered by HCFA. Thus, precise estimates are available that indicate the degree of variation in resource consumption among these groups. The use of DRGs for predicting mortality or other clinical outcomes was never intended in their construction and has not been validated.

* Importance of Findings. Cost assessment and resource utilization are objectives for which DRG classification is useful.

* Limitations. Simply because resource consumption may often correlate with adverse consequences of care, it is not wise to assume that a system based on costs can be applied to clinical outcomes. It is worth noting that HCFA's first mortality model was based heavily on DRGs and was quickly abandoned in favor of custom-tailored risk adjustment.

Acuity Index Method (AIM)

* Purpose and Development. Designed to reduce the variance within DRGs to enable more precise cost analysis, AIM was created using physicians' judgment and a database of 10 million patients discharged from California hospitals.

* Data Collection. Because no data collection is required beyond the discharge summary, AIM can be applied to state or any other publicly available data sets.

* Techniques. DRGs are subclassified into five levels of severity that predict different lengths of stay. Client hospitals purchase a service contract from the developer to compare their charges, utilization, and mortality rates to the statistical norms derived from AIM's database for each level of AIM severity within each DRG.

* Experience. Although its applicability is as wide as DRGs', AIM has had comparatively little use. Its owner, Iameter (San Mateo, Calif.), had 20 client hospitals in December 1988. [3] No revisions have been announced since AIM's inception three years ago.

* Cost. Annual cost is moderate, depending on the terms negotiated for the service contract. No additional personnel, computers, or data collection are required.

* Validity. Selected cases have been cited at sales presentations and seminars, but no independent, systematic, aggregate analysis has been performed that substantiates the validity of the system.

* Importance of Findings. Iameter's anecdotal material illustrates quality problems that have been identified by applying AIM's adjustments to potential quality monitors, such as mortality. However, like DRGs, AIM was developed as a key to resource consumption, and we have seen no scientific evidence to support its use for quality assessment.

* Limitations. AIM has not had the benefit of extensive study and refinement as have HCFA's mortality data and the DRGs. There are no indications of revisions being undertaken to incorporate the experience of even the small number of users, and there is no objective evidence that AIM's categories are any better than updated DRGs.

Patient Management Categories

(PMCs)

* Purpose and Development. PMCs were developed by consensus panels of physician experts at Pittsburgh Research Institute, an affiliate of Blue Cross of Western Pennsylvania, to focus on resource consumption.

* Data Collection. Only the standard discharge summary is needed. Because the system requires only minimal discharge data, comparisons can be made with large databases from public or other sources.

* Technique. The system assigns to each patient a category that is related to the cost of managing patients in that category according to a clinically suitable standard. Standards specify types and amounts of diagnostic and treatment services and expected lengths of stay in special units of the hospital as well as general length of stay. Thus, actual resource use can be compared with expected use for each Patient Management Category. The first component of the system, classification of patients, can be used separately from the management paths and cost weights. Menu-driven functions provide reports on frequency distribution and averages, geometric mean length of stay and outlier length of stay and thresholds, case mix index calculations, analyses of comorbidity, and query-driven patient abstract displays. PMCs are completely automated for personal computers as well as for mainframes.

* Experience. PMCs currently are used in comparative cost analyses within and among hospitals and in several projects with payers and business coalitions conducting analyses of hospital costs. Some three dozen users have been reported.

* Cost. Software costs are moderate.

* Validity. A comparative study [4] indicated that PMCs were better than other systems in predicting costs for the broader adjacent DRG categories. However, like DRGs and AIM, PMCs were not originally designed to predict clinical outcomes, and the validity of the patient classification system and the patient management paths bor this purpose has not, to our knowledge, been substantiated.

* Importance of Findings. The system is useful in resource utilization and cost comparisons. Although its mortality rates have a clinically more specific basis than those of DRGs, they were not designed for quality assessment and, consequently, could not be expected to be as reliable as HCFA's mortality data.

* Limitations. No justification exists for using PMCs for other than their originally designed purpose.

Disease Staging

* Purpose and Development. The concept kf staging was first developed for cancer patients, for whom it seemed useful to trace the increasing burden of disease as it would progress without treatment from a mild, localized stage to death, the final stage. The original idea was to develop clusters of similar patients for comparative studies. Because increasing sums of money are spent as a patient's condition worsens, staging now is used to anticipate the costs of patient care by associating them with severity of illness.

* Data Collection. A computerized version, Coded Disease Staging, uses the discharge summary. There is also a manual version, Clinical Disease Staging, which requires clinical findings and test results from the patient's medical record.

* Technique. Four major stages are defined for each disease; as many as 12 diagnoses for each patient may be staged. A recent modification, the Q-Scale, provides a general measure of severity for the patient. It is based on regression analyses of each disease staging body system for a database of nearly 7 million discharges from hospitals in various regions of the United States. A patient's expected resource use also can be compared with the national average use within the same DRG. A score of 100 is equal to the national average for the same DRG; 150 indicates a patient whose severity of illness is 50 percent higher than the national average for that DRG. In both cases, Q-Scales measure dollars and ratios of dollars and can be averaged for comparative analyses of resource consumption.

* Experience. About 300 hospitals use Coded Disease Staging and the Q-Scale, and Maryland and California use it to analyze statewide UB-82 claims databases. The developer supports its comparative analyses with a database of 26 million medical records.

* Cost. The cost of computer software is moderate. Staff time is required to obtain data from the medical record for the manual version.

* Validity. The system's basic assumption that resource utilization is a proxy for severity of illness has not been proven. SysteMetrics' own studies have focused only on the Q-Scale's cost-predictive ability and have resulted in claims that the Q-Scale increases the explanatory power of DRGs by 50 percent. A 1986 report [4] indicates that the coding of disease staging was not consistent when diagnoses were reabstracted, and severity assignments differed substantially as a result. In comparing findings of several severity systems with those of a physician panel, the same study found that Coded Disease Staging had the lowest correlation with physicians' scores.

* Importance of Findings. Quality of the input data is always a factor in severity systems, particularly when subjective judgment is involved and when numbers of patients are not large. As a cost-predictor, the Q-Scale is similar to refinements of the DRG; the general Q-Scale is considered by some to be a more accurate predictor of costs than Q-Scale comparisons within DRGs.

* Limitations. From a clinical standpoint, the association of specific stages of illness with incremental risk of particular outcomes has not yet been clearly established. Consequently, the developer (SysteMetrics, Santa Barbara, Calif.) does not intend use of the system for outcome predictions.

PRAGmatic System

* Purpose and Development. This new system was designed to focus on quality, not resource consumption. Its Patient Risk Adjusted Groups were developed to predict mortality, nosocomial complications, and unusually long or short lengths of stay. The system was derived empirically from a nationally representative sample of hospitals containing approximately 800,000 discharges drawn from statewide data sets and the expanded, modified Medpar file. Unlike HCFA's database, which is limited to Medicare patients, PRAGmatic's database includes a cross-section of patients.

* Data Collection. The use of only readily available administrative data facilitates comparisons among hospitals.

* Technique. Patient Risk Adjustment Groups are designed to be mutually exclusive, clinically discrete, and homogeneous, based on variations in predicted rates of outcomes used to monitor quality of service. Patient data are grouped and then compared with risk-adjusted norms for each group. Clients can specify how groups are aggregated for comparative analyses. The owner, Corporate Cost Management (Gaithersburg, Md.), performs analyses on its mainframe computer and supplies clients with standard summary reports that include differences between observed and expected values for each outcome indicator in each disease category.

* Experience. Development and programming of the system were completed early in 1989, and pilot testing has been conducted in a small number of hospitals.

* Cost. Because the system is not currently available for installation in individual hospitals, no staffing or computer equipment is required. Standard summary reports are supplied at moderate cost.

* Validity. Mortality and complication rates are direct indicators of effectiveness of care, while hospital length of stay can be prolonged when treatment is ineffective or harmful. Although developmental tests have validated the PRAGmatic groupings, no independent or comparative studies are available to verify the system's performance capabilities.

* Importance of Findings. The PRAGmatic System's comparison of actual occurrences with empirically derived norms enables identification of areas where performance is higher or lower than expected. Aggregation/disaggregation capabilities in grouping maximize inferential power for low volumes of cases. The system goes beyond HCFA's mortality release in its use of complications and lengths of stay as additional outcome monitors. While not intended to predict or measure resource utilization, it can be used, along with a system designed to predict resource consumption, to indicate the effectiveness portion of the cost-effectiveness equation.

* Limitations. As with other systems using only administrative data, grouping can be affected by transcribing and coding errors and is limited by the lack of specific physiologic information. Using only administrative records, it is difficult to distinguish between complications and comorbidities. The PRAGmatic system should not be confused with those designed to predict resource consumption.

References

[1] Medicare Hospital Mortality Information, 1987. Washington, D.C.: Superintendent of Documents, U.S. Government Printing Office, 1988.

[2] Pine, M., and others. "Potential Effectiveness of Comparative Casemix-Corrected Adverse Outcome Rates as Quality Monitors in a Simulated Hospital System." Medical Decision Making 9(2):104-15, April-June 1989.

[3] Gardner, E. "Research Team Studying Six Severity of Illness Systems." Modern Healthcare 18(51):26-7, Dec. 16, 1988.

[4] Thomas, J., and others. "An Evaluation of Alternative Severity of Illness Measures for Use by University Hospitals." Department of Health Services Management and Policy, School of Public Health, University of Michigan, Ann Arbor, Dec. 1986.

Michael Pine, MD, MBA, is a Research Associate in the Department of Medicine at the University of Chicago and President, Michael Pine and Associates, Inc., Chicago, Ill. Dr. Pine is an Associate Member of the College's Forum on Quality Health Care. David W. Smith, PhD, MPH, is Executive Vice President, Data Analysis and Systems Design, Michael Pine and Associates, Inc., Chicago, Ill.