An overview for medical directors - part two.

An Overview for Medical Directors--Part Two

Severity-of-illness systems have been created that focus on clinical outcomes but that require the collection of information that is not on the billing form. Included in this group are MMPS, APACHE and its variations, MedisGroups, and CSI. Each of these is detailed below, using the same set of criteria that was used for the administrative data based systems.

Clinically Based Systems

Medicare Mortality Predcitor

Systems (MMPS)

* Purpose and Development. MMPS was designed by HCFA to assist hospitals in pursuing potential problems identified by the HCFA mortality release. It focuses on myocardial infarction, stroke, pneumonia, nad congestive heart failure. A sample of nearly 6,000 Medicare patients was used for validation.

* Data Collection. Clinical data as wel as administrative data must be collected.

* Technique. MMPS predicts the probability of death in four categories of disease that are common and frequently lethal among Medicare patients. Users can obtain more specific information than from the general HCFA mortality release by comparing their hospital's death rates for each of these diseases to those for the 6,000 patients in the normative database. The system is in the public domain but requires a personal computer.

* Experience. Because the system is not owned commercially, it has not been aggressively marketed and has had relatively little use.

* Cost Software is relatively inexpensive.

* Validity. MMPS supplements HCFA's risk adjustment technique by incorporating clinical information that bears on hospitalized patients' risk of dying and is not available in administrative records. Although it developed from a small sample of patients, careful statistical techniques were used to derive accurate estimates of risk.

* Importance of Findings. MMPS permits hospitals to predict mortality in four diseases that contribute substantially to overall mortality in Medicare patients.

* Limitations. The system include only four diseases and can't be directly related to HCFA's 17 disease categories. Therefore, the amount and usefulness of supplemental information obtained by the analyses is relatively small.

APACHE II 

* Purpose and Development. APACHE II is a refinement of the original APACHE system designed at the George Washington University Medical Center to make prognoses about patients that would help defined appropriate admission criteria for ICUs. APACHE II was developed by an expert panel and was validated by a sample of nearly 6,000 patients in 19 ICUs at 13 diverse hospitals. It provides a risk index for ortality among ICU patients and predicts each patient's length of stay in the ICU.

* Data Collection. APACHE II requires a modest amount of physiologic information along with limited demographic and diagnostic data.

* Technique. Each patient is assigned an "acute physiology score" that is derived from physiological measurements weighted according to their prognostic importance. Points are added for patient age and presence of chronic disease.

* Experience. APACHE is one of the older systems. Its first version, introduced in 1981, was based on extensive investigation. APACHE II, introduced in 1985, currently has more than 200 users. The system has been modified and combined with other measures to broaden its applicability. APACHE IIB applies the APACHE system to the four MMPS categories in an effort to broaden usage to the entire patient population. To evaluate efficiency, APACHE may be combined with the Therapeutic Intervention scoring System (TISS), which gauges resource consumption by using expertly assigned weights to grade therapeutic, diagnostic, and monitoring tasks in the ICU. The Simplified Acute Physiology Score (SAPS) was an independent attempt at simplification that reportedly attained results roughly comparable to APACHE II. The Sickness Score (SS) is the United Kingdom's version; it has only inor modifications and similar predictive power. The Physiology Stability Index (PSI) was developed for pediatric ICUsf an independently updated version, PRISM, was independently validated in one study with excellent results. MPM is a modification of APACHE II in which the predictor variables were selected by computer search. MPM uses a checklist of 11 items to evaluate mortality risk at the time a patient is admitted to the ICU and at 24 and 48 hours after admission. Initial validation tests showed no statistically significant differences between the two systems. APACHE II performed significantly better in a more recent independent comparison, but definitive conclusions would be premature. The extent to which all these variations are used is not known, but if imitation is the sincerest form of flattery, APACHE II appears to have earned respect. APACHE III, with more homogeneous disease categories and risk estimates developed from a nationally representative database, is under development.

* Cost. APACHE II's scoring system and prediction formula are in the public domain ad can be programmed readily for a personal computer. The software is moderately priced.

* Validity. APACHE II has undergone extensive testing and statistical development and validation.

* Importance of Findings. APACHE II lrovides excellent prediction of patient-related risk of death after admission to an intensive care unit. It also has proven useful in providing objective criteria to assist in admitting patients to and discharging them from intensive care units.

* Limitations. APACHE II is a well-developed and well-validated risk adjustment system whose applicability currently is limited to intensive care units. It is less useful for patients in burn units, for whom more specialized measures are available. Despite APACHE II's proven validity as an epidemiological tool, its developers still do not consider their system's current patient groups refined enough to guide clinical decisionmaking for an individual patient, and they caution that their system should be used along with other evidence. Neither APACHE nor APACHE IIB has sufficiently wide applicability to be used for hospital wide quality monitoring.

MedisGroups

* Purpose and Development. MedisGroups developed from an identification by clinicians of clinical findings associated with the failure of the body's major organs, independent of diagnosis. Patients are categorized into severity groups on the basis of specific clinical findings observed within 48 hours of admission. They are categorized into morbidity groups on the basis of findings observed after 48 hours. Findings that emerge from sophisticated technology and equipment are favored. The original concept of diagnostic independence has been revised to include the use of DRGs in categorization.

* Data Collection. Using the system requires collecting a wide variety of clinical data elements from among 260 Key Clinical Findings. The data are gathered from medical records and the discharge summary and are entered into a computer mainframe of sophisticated PC or are sent to the manufacturer's service bureau for analysis.

* Technique. A predetermined severity score is assigned by the computer to each data element. A proprietary algorithm then places the patient in an kvarall admission severity group that is not specific to diagnosis. Morbidity, or lack of response to treatment, is evaluated by a second set of findings obtained during a specified interval after either admission or operation. Many different kinds of reports may be generated by the system, including mortality rates, morbidity rates, and severity scores for individual facilities.

* Experience. MedisGroups claims more than 400 users and has been mandated for hospitals in the state of Pennsylvania. It now has a database of as many as 1.5 million cases. The system has been aggressively marketed, and its developers have undertaken repeated revisions in response to comments by users and the perceived needs of potential users.

* Cost. Software is relatively expensive. At least one full-time equivalent nurse or medical records technician is required for data collection, and additional assistance in computer use and interpretation of the findings also is helpful, because the system is very complex.

* Validity. Because MediQual Systems (Westborough, Mass.), the developer, closely guards the MedisGroups' algorithm, few users of researchers have had the opportunity to study it. A group of Boston University researchers concluded that the data elements were clinically meaningful and comprehensive and weighted in a reasonable manner. They thought the algorithm was acceptable and similar to that of other severity rating systems. [1] MedisGroups' predictions of organ failure had a high correlation with those of the University of Michigan's expert panel. [2] However, the Boston University group found that MedisGroups alone explained only 3.3 percent of cost variation, compared with 51.9 percent for DRGs, and only slightly (3 percent) improved the predictive power of DRGs. This group also found that, while admission MedisGroups scores might improve predictions of inhospital mortality, small numbers of key clinical findings provided more powerful predictive capacity. [3]

* Importance of Findings. MedisGroups' original design as an index of risk of failure of major organs makes it appropriate as a tool for monitoring some aspects of quality of care. Key clinical findings and individual severity ratings can supplement administrative data to refine estimates of the probability of specific adverse outcomes.

* Limitations. While MedisGroups represents a major investment and produces large volumes of data, it also can be confusing to users. Current reports do not adequately adjust predicted outcome ranges for risk, and aggressive analyses of small subsets of cases may result in many erroneous conclusions. While value has been found in exposing clinicians to severity and outcome data, Medigroups' complex, rapidly changing yet analytically inflexible proprietary system may often fail to provide clear insights for clinical and managerial decisions. MedisGroups' morbidity predictors have not been demonstrated to clearly differentiate cases where abnormal laboratory values are genuine indications of worsening condition from those in which they merely are consistent with normal expectations during the course of treatment. MedisGroups has the same problems as any system that attempts to monitor quality with a system that was not designed for that purpose.

Computerized Severity Index (CSI)

* Purpose and Development. CSI was developed by researchers at Johns Hopkins Univesity to measure severity for the purpose of explaining variation in resource consumption. The system was devised as a successor to the Severity of Illness System and is based on a team of clinical experts' determinations of severity of patients' characteristics and analysis of a previous severity database containing more than one million patient abstracts.

* Data Collection. Clinical data must be abstracted from the patient's medical record.

* Technique. CSI begins with the patient's diagnosis and adds clinical findings (e.g., lab results, vital signs, and data from the history and physical) that contribute to the patient's physiologic burden of illness. Branched logic is used to limit the number of clinical data elements collected for each patient from a potential 606 to an average of 32. Severity is categorized for each condition; then a general severity category is calculated on the basis of the principal diagnosis, the most severe secondary diagnoses, and the diagnoses' interaction.

* Experience. More than five dozen users have been reported.

* Cost. Costs are relatively high. Installation, training, software license fee, and staff time in collecting the data and operating a personal computer all must be considered.

* Validity. CSI is too new to have been evaluated in the University of Michigan study, and independent information about its reliability is not available. A validation study of the system's ability to predict cost and length of stay recently was completed in New Jersey. [4] It involved 85,000 cases from 25 hospitals, and the results have not yet been published. CSI's owner, Health Systems International, has made its severity criteria available to users for evaluation. The system's ability to predict outcomes has not been validated.

* Importance of Findings. Severity scores at various points of a patient's stay can be compared to assist in assessing treatment progress. CSI's severity scores also may improve upon the accuracy of DRGs in predicting costs. Because information is collected on the basis of diagnosis, it can be used to check the medical record and ICD-9-CM coding.

* Limitations. While CSI utilizes clinical information not available in administrative records, the extent to which its algorithms improve predictions of adverse outcomes based on less expensive risk adjustment techniques has not been determined. Data needed to evaluate the system's effectiveness and perfect its internal logic still are being acquired.

Weighing Alternatives

In weighing alternative severity systems, it is important to realize that none of them overcomes the effect of random variation. It also should be noted that many fine hospitals have been overwhelmed to the point of inaction by the sheer volume of data that can flow from some of these systems. Even if they had the staff to cope with it, they wouldn't be sure what to look for or what the findings mean. Before spending money that will produce reams of computer printouts, be sure you can manage the results and that they are likely to make a difference in the way things are done.

Marketing strategies have packaged these severity systems with, or offered them in the context of, management information systems, data systems, quality assurance systems, information or report services, consulting services, etc. We have attempted to described or discuss only the risk adjustment components here, because we believe they should stand on their own merits. Just as you wouldn't buy an automobile for its clock or radio, you shouldn't acquire a severity system because it is linked to some other system or service--unless you have determined that you really need that other component and cannot get a superior version of it separately. Otherwise, you may find yourself with problematic or superfluous features you would not have had to accept had you considered each item individually.

A final word of advice is not to lose sight of your clinical and managerial goals. As a medical director, your purpose in assessing the outcomes of care is to gain insights that will improve clinical and management decisions. The best that a packaged system or methodology can do is process data and provide reports. The rest is up to you.

References

[1] lezzoni, L., and others. "MedisGroup's: A Clinical and Analytical Assessment." Health Care Research Unit, Section of General Internal Medicine, Boston University School of Medicine, July 1, 1987.

[2] Thomas, J., and others. An Evaluation of Alternative Severity of Illness Measures for Use by University Hospitals." Department of Health Services Management and Policy, University of Michigan, Ann Arbor, Dec. 1986.

[3] lezzoni, L., and others. "The Ability of the MedisGroups and Its Clinical Variables to Predict Cost and In-Hospital Death." Research report to HCFA under agreement No. 18-C-98526/1-04. Health Care Research Unit, Section of General Internal Medicine, Boston University School of Medicine, July 1, 1988.

[4] Averill, R., and others. "A Study of the Relationship between Severity of Illness and Hospital Cost in New Jersey Hospitals." Report of study performed by Health Systems International for the New Jersey Department of Health, 1989.

Michael Pine, MD, MBA, is a Research Associate in the Department of Medicine at the University of Chicago and President, Michael Pine and Associates, Inc., Chicago, III. Dr. Pine is an Associate Member of the College's Forum on Quality Health Care. David W. Smith, PhD, }PH, is Executive Vice President, Data Analysis and Systems Design, Michael Pine and Associates, Inc., Chicago, III.