Red herring or horse of a different color?
Recently, interest in measuring severity of illness has reached new heights, with at least one state Pennsylvania) requiring such measurements. Proponents have claimed that severity measurement can be used for a variety of purposes, including adjusting outcome data, measuring quality of care, and calculating payment for care. Despite current the interest in severity measurement, considerable confusion exists about the concepts underlying severity measurement and its use.
This article provides a framework for understanding the output of health services and factors affecting the production of health. Within this context, the paper examines the concept of severity of illness and its utility for assessing provider performance (in terms of assessing the effectiveness of care; adjusting outcomes of care, e.g., mortality rates; and assessing the quality of care) and paying for medical care.
The article deals only with theory, not practice. While such issues as validating diagnoses and otherwise ensuring the integrity of data are critical to the evaluation of measurement systems, the magnitude and complexity of these issues, put them beyond the scope of this work. The article does not describe or evaluate the various severity measurement systems currently available, which also has been done previously.2 Rather, it provides the conceptual basis for deriving criteria and some suggestions for their evaluation.
Measuring Health Outcomes
A person's health status is the result of four sets of interrelated factors: (1) biology (genes or genetic programming); (2) behavior; (3) the pre- and postnatal environment, encompassing the physical (e.g., climate, pollution), biological (e.g., viruses, bacteria), economic (e.g., food, shelter, clothing), and social (e.g., population aggregation, workplace factors, social support systems) environments; and (4) the health system. Health status is a measure of a person's health throughout life and includes the number of years lived and the health quality of life, taking into consideration morbidity, institutionalization, and functional ability.
Patients seek health care because they believe it will maintain or improve their health. The provider must (1) diagnose the problem, (2) assist the patient in selecting among alternative interventions by providing information on each alternative, including the patient's prognosis without any intervention and that likely to result for each alternative; (3) implement the selected intervention; and (4) followup to ensure the intervention worked as intended, to take care of side effects, or to suggest additional interventions. The final choice depends on patient acceptance of the intervention.
Health outcomes, or the patient's postintervention health status, may be viewed as the result of three sets of variables.3
The patient's preintervention health characteristics.
The intervention, encompassing the physician's ability to categorize the patient for treatment selection purposes, to select the best treatment for the patient, and to implement the selected treatment, including recognizing and managing any resultant untoward outcomes.
The environment in which the intervention takes place.
The relative impact of any one of these sets of variables on outcome will depend on the type of treatment under study and on the time at which the study was conducted.
Moreover, the importance of interactive effects may also vary dynamically.
Ideally, a physician's choice of diagnostic and therapeutic interventions would be based entirely on substantiated scientific information. Practically, however, choices must be based largely on training, experience, or assumption because of the dearth of scientific substantiation of the effectiveness of interventions.4 Moreover, health outcomes should be measured in terms of the patient's future health status.5 However, virtually no information exists about such outcomes.6 In fact, there is a dearth of any type of information on outcomes7. In practice, when measured at all, outcomes are usually expressed as 5-year survival or mortality rates, or even less satisfactory measures. Mortality is an inadequate measure of outcome, because most interventions are likely to alter the quality, and not necessarily the quantity, of life.8
Patient Variables; Diagnosis
Correct diagnosis is key to estimating the course of the patient's disease and to selecting the best treatment. Moreover, diagnostic classifications are used increasingly for administrative and other purposes, adding to their importance. The physician faces uncertainty and risk in placing the patient in the correct diagnostic category. Additional tests, for example, may reduce the risk of categorizing the patient incorrectly, but their cost or inconvenience to the patient may overcome the perceived benefits that would flow from increased diagnostic certainty.
This view of diagnosis conceives of a large, but finite, number of categories into which a physician may place a patient for prognostic or therapeutic purposes. A lack of consistent criteria results in fuzzy rather than sharp classifications. Moreover, differences among providers regarding concepts and criteria may result in variations in diagnostic classification. For any given classification, research could produce solid information on outcomes with and without intervention. Given the magnitude of possible diagnostic classifications and their lack of standardization, it is not surprising that few such specific outcome data exist.
Diagnosis is complex and results in classifying the patient according to medical and sociodemographic factors. Medical factors consist of:
Disease, classified by
*Derangement, e.g., cell mechanism, organ system.
*Producer, e.g., pathogenic organism, chemical, physical force.
* Recognizable characteristic diagnostic manifestations.
Disease may be defined as any departure from health. If one considers health and normal physiology to be synonymous, disease represents deranged physiology. Thus, health is the outcome of normal (in contrast to deranged) physiology. Ultimately, health is defined relatively, either by desired or expected functional abilities.
Deranged physiology may be produced by the patient's genetic make-up or by environmental factors such as pathogenic organisms, chemicals, forces resulting in injury, or their interaction. It may result in various diagnostic manifestations. Patients may vary in their physical, mental, and emotional reactions to these manifestations and symptoms may vary. Thus, a patient's ability to function and his behavior, including decisions to seek care, are influenced not only by the deranged physiology and its producers but also by his/her responses to its manifestations, which are in turn affected by genetic make-up, experience, environmental variables such as culture, and interactions among these factors.
A disease stage represents a recognizable place in a temporal journey from health to death along a defined route. For example, a disease's natural history may be divided into two or more stages. This concept is best known in oncology but has been applied to all diseases.9 Stage also implies irreversibility. Having progressed from Stage I to Stage II, one can never again be labeled Stage 1. The prognosis for a treated Stage 11 patient may be the same as for an untreated Stage I case, but the patient has not reverted to Stage I. Rather, the patient has progressed to "Stage II treated X months ago with intervention A." Thus, previous therapy is another important diagnostic classification aids.
Often, patients have several diseases. At any given point, attention focuses on one, of these disease entities; the rest are referred to as comorbidities. Each disease may be classified by type, stage, and previous therapies. Specific combinations of diseases, including their stages and previous therapies, may represent unique medical diagnostic categories for purposes of prognosis and therapeutic intervention.
Finally, a patient must be classified by sociodemographic and other factors that influence outcomes or treatment selection, including, for example, age, income, living arrangements, personality type, and medical insurance coverage.
The number of categories to which a patient could be assigned is vast, if not infinite. Because the number of therapies is often limited, the number of diagnostic categories relevant to outcomes is usually larger than that for treatment selection. The number of relevant categories is limited by the state of knowledge, as is the selection of appropriate therapy and expectations of outcome with and without treatment.
Intervention Variables: Therapy
Having assigned the patient to a specific diagnostic category, the physician can select among alternative interventions. The alternatives may be many or few, depending on the diagnosis. The alternatives may be a subset of all possible alternatives and maybe predicated on the physician's training or other factors, such as the availability of equipment or services. Surgeons, for example, may favor surgery while internists may favor other interventions. Such preference would not matter if the outcomes of the interventions were the same. Practice style, or technically permissible practice variations, may be defined as favoring one set of interventions over another when the health outcomes of all sets are essentially equal. However, they may persist even if the outcomes are unequal because the physician is skilled in only one set of alternatives. Where an alternative in which the physician is not skilled is dearly indicated, the physician might refer the patient where such referral is practical and in the patient's best interest.
The physician's chosen course of action, the therapeutic regimen, consists of one or more interventions, each implemented in a prescribed manner. An intervention comprises two parts: a technology and its delivery mechanism. The technology is the active ingredient that produces the desired effect. In some cases, e.g., antibiotics, the delivery mechanism influences outcomes relatively little. In others, the delivery mechanism is the technology, e.g., psychotherapy. With respect to defining the regimen, both intensity of intervention and intervention intervals are of interest. Intensity is a product of dose and number of repetitions (dose rate times duration). The effectiveness of some interventions, e.g., surgery, are little or not at all influenced by external factors, e.g., patient behavior. Others, e.g., drug regimens, are greatly influenced, e.g., by the patient's willingness to take drugs as prescribed.
Given the correct regimen, outcomes depend on the skill of the physician and others in implementing interventions. For example, surgeons performing a given procedure often may do it better than those performing it infrequently. Further, some hospitals may offer better support, e.g., nursing care, than others, or physicians may behave differently when working in different hospitals, resulting in different outcomes to patients even if operated on by the same surgeon.
For any disease, there may be several alternative therapies. Further, for any therapy there may be different regimens. Moreover, implementation of each regimen may produce various states, depending on patient behavior. Thus, patients' treatments may represent a vast number of therapeutic classifications. The potentially large number of relevant therapeutic possibilities, especially when coupled with the large number of relevant diagnostic categories, makes evaluating therapeutic effectiveness quite difficult.
In some cases, the impact of environmental variables may be reflected in the patient's diagnostic classification. For example, the health outcomes of an intervention may depend on income. Thus, patients may be classified according to income for purposes of selecting therapy or adjusting outcomes. In other cases, their impact cannot be reflected in patient classification, and some other type of adjustment is necessary. For example, if employment is an outcome of interest, an area's unemployment rate for the type of patient may be a more important determinant of outcome than the intervention. Similarly, the type and extent of air pollution may be a determinant of the outcome of therapies for respiratory ailments.
In measuring an intervention's outcomes, or determining its effectiveness, one must either assume that environment variables have little or no affect on outcome or, if they do, that they influence alternatives equally. When this is dearly not the case, adjustments may be needed. Because the principle of compensating for environmental variables parallels that for patient variables, it will not be considered further.
Measuring intervention Effectiveness
Treatment potency is a measure of an intervention's ability to maintain or restore a patient's health (see figure 1, page 9). For practical reasons, a patient's health status after treatment can be compared to the average for people of the same age and sex or those who do not have the health problem (or who were not admitted to the hospital, for example). Given this normative definition of health, one can conceive of treatments as providing "superhealth," a health status greater than average. To the extent that everyone availed themselves of such preventive or therapeutic interventions, the average could not be exceeded for patient populations, but could be by an individual. Treatment potency can be quantified as the percentage of health loss that is recouped by intervention. These data are generally not available presently. Instead, interest has focused on treatment effectiveness.
Effectiveness is the extent to which intervention is superior to no intervention in maintaining or improving health. it is the difference in outcomes between intervention and the course of the disease if untreated see figure I . Often it is not possible or permissible to leave patients without treatment. The result of studies in such situations is the marginal effectiveness of one treatment over another, for example of an experimental treatment over the usual treatment. If knowledge of natural outcomes is lacking, which is usually the case, one can easily overestimate therapeutic effectiveness, as was the case with the use of penicillin to treat for syphilis.10
The most effective intervention is the one that produces the greatest health status gain for the patient. Because variability (not certainty) characterizes the health outcomes of interventions, an intervention's effectiveness depends on observing its effects on a population of patients. As a group, patients are more helped than harmed by an effective intervention. If there are no known characteristics to divide patients into those more likely to be helped than harmed, there is an inherent risk in the intervention. Further, just because a patient's observed health status is worse after the intervention does not mean that the intervention was ineffective. The patient's health status may have been even worse without the intervention.
The best knowledge of the effectiveness of interventions comes from scientifically designed and well-controlled trials. In such trials patients are assigned randomly to treatment and control groups, and preferably neither patient nor provider knows the assigned group (double-blind). Further, the assessment of effectiveness is conducted by an impartial third party (triple blind trial) to eliminate the inherent bias of the person who invented or is a proponent of the intervention. When an intervention is implemented under ideal conditions, the resultant effectiveness is termed efficacy, a special case of effectiveness that defines the intervention's maximum expected health benefits for the specified conditions.
Determining the effectiveness of interventions is one of the functions of health services research." To date very little money has been spent on health services research.12 Moreover, according to one evaluation, despite researchers' best intentions, most studies are invalid or scientifically inadequate in terms of their design, data, statistical inferences, or documentation.13 Consequently, we know scientifically little or nothing about the effectiveness of most interventions.
Researchers go to extraordinary lengths to ensure valid measurement of the effectiveness of interventions. The objective is to control or account for all variables that might influence outcome and to be able to judge differences in outcomes observed between experimental and control groups. Figure 2, page 10, compares this experimental situation with that confronted by someone judging a provider's performance. As Figure 2 shows, valid measurement of provider performance is a daunting task. Given researchers' lamentable performance in assessing effectiveness, one cannot be sanguine that provider performance can be measured well, if it can be done at all.
In Part 2 of this article in the September-October 1989 issue of Physician Executive, the author will begin a discussion of the specific issues that surround the measurement of provider performance and the assessment of quality in practice.
1. Goldschmidt, P. "The Appropriate Organizational Locus for Constructing Indicators of the Quality of Hospitals and Physicians and for Evaluating the Validity of Those Indicators." Contractor document prepared for U.S. Congress, Off ice of Technology Assessment project (The Quality of Medical Care: Information for Consumers, OTA-H-386. Washington, DC: U.S. Government Printing Off ice, 1988).
2. An Evaluation of Alternative Severity of Illness Measures for Use by University Hospitals: Volume Management summary, Volume 11,Technical Report. Ann Arbor, Mich.: Department of Health Services Management and Policy, School of Public Health, University of Michigan, 1986.
3. Goldschmidt, P. "Cost-effectiveness Model for Evaluating Health Care Programs; Application to a Drug Abuse Treatment." Inquiry 13(l):29-47, March 1976.
4. U.S. Congress, Office of Technology Assessment Assessing the Efficacy and Safety of Medica/ Technologies, OTA-H-75. Washington, D.C.:
U.S. Government Printing Office, 1978.
5. Goldschmidt, P. A Model for Measuring the Health Status of a Population: Application to the United States. Baltimore, Md.: Policy Research inc., 1978.
6. Najman, J., and Levine, S. "Evaluating the Impact of Medical Care and Technologies on the Quality of Life: A Review and Critique." Social Science and Medicine 15F: 107-15, 1981.
7. Williamson, J., and others. "The Quality of Medical Literature: An Analysis of Validation Assessments." In Medical Uses of Statistics. Boston, Mass.: NEJM Books, 1986.
8. McDermott, W. "Absence of indicators of the Influence of Its Physicians on a Society's Health: Impact of Physician Care on Society." American Journal of medicine 70(4):833-43,april, 1981.
9. Gonelia, J., and others. "Staging of Disease: A Case-mix Measurement." JAMA 251(5):637-44, February 3, 1984.
10. Gjestland, T. "Oslo Study of Untreated Syphilis: Epidemiologic Investigation of Natural Course of Syphilitic infection based upon a Restudy of the Boeck-bruusgaard Material." Acta Dermato-Venereologica (Supplement 34-35: 1 368, i-Ivi, 1955).
11. Goldschmidt, P. "Health Services Research and Development: The Veterans' administration Program." Health Services Research 20(2):789824, June 1986.
12. Institute of Medicine. Assessing Medical Technologies. Washington, D.C.: National Academy Press, 1985.
13. Williamson, J., and others. Medical/practice Information Demonstration Project. Final Report. Baltimore, Md.: Policy Research inc., 1979.
T H E A U T H O R
Peter G. Goldschmidt, MD, DRPH, DMS, is Vice President for Research and Development, Quality Standards in Medicine, inc., Bethesda, Md. Address inquiries to: Peter G. Goldschmidt, MD, DRPH, DMS, 5101 River Road, #1913, Bethesda, Md., 20816.
|Printer friendly Cite/link Email Feedback|
|Title Annotation:||part 1; measuring severity of illness|
|Author:||Goldschnidt, Peter G.|
|Date:||Jul 1, 1989|
|Previous Article:||Long-term care - a growing employer concern.|
|Next Article:||Certification: goals set, goals met.|