The use of imperfect data in managed care organizations. (Imperfect Data).

AS MEDICAL MANAGEMENT INFORMATION SYSTEMS continue to evolve, massive amounts of data are rapidly being accumulated. This wealth of information presents mind-numbing possibilities for creating reports and graphs at various levels of complexity. The temptation for an analyst to generate "too many reports" needs to be quelled, to prevent presenting a bewildering array of information to medical professionals. The large volume of data, while impressive, is also frequently characterized by a host of limitations that become the focus for detractors of the change process.

Two critical milestones appear to be occurring in the development of medical groups moving to improve medical care effectiveness. These include the ability to work with imperfect and unflattering data. There is a clear linkage between these two concepts, because forward clinical improvement or business planning is often delayed as individual physicians seek to await "perfect data" when confronted with unflattering information, In the form of "profiles" in particular, providers often react negatively, with complaints that the information is "imperfect" or that it fails to capture some nuance of their sicker or unique patient populations.

The translation of imperfect information to effective clinical practice anyway remains a success fundamental to managing highly competitive medical groups and health plans. It is centrally dependent on the understanding, use, and application of "imperfect data."

Imperfect data (please see Figure 1) is easily characterized by analytical purists and includes information that:

* Is incomplete or unfinished. Database records frequently do not have entries for all individual statistics required. As a result, reporting must, of necessity, be performed on population samples, rather than on the whole of the desired information base.

* Consists of incomplete individual record components, with holes or absent portions of records. For whatever reason, there is a maddening tendency in health care databases to have incomplete portions of records. Data entry problems, incomplete cooperation of patients, or simple mechanical glitches often result in this condition.

* Is characterized by areas of local quality deterioration. An example would be when 700.0 may be entered instead of 70.0.

* Is fragmented across time, such as data collected over the course of a year. Claims-based data typically have a three-month "IBNR" or Incurred But Not Reported runoff, making current real time information subject to approximation or final results requiring a wait of three months for all of the information to be collected.

* Is believed to be nonspecific or insufficient to a desired use. For example, drawing quality outcome indicators out of the paid claims database.

* Consists of non-normal distributions, making simple analyses or analyses which use assumptions of normal distributions more difficult. For practical purposes, removing "long tall" outliers typically seen on some health care distributions allows the distributions to then approximate a normal curve.

* Is absent of clinical severity or complexity adjustment indicators. This is often a rallying cry for physicians or other providers who are identified as being clear over-utilizers. Physicians and hospitals will frequently explain outlier status, even highly impressive outlier status, on the basis of the fact that they are atypical. They report this must be true, since because of their perceived high quality, they attract sicker patients. While this possible effect is undeniable, the effect becomes less as larger patient populations are assumed, and is invoked much more often than is truly reflected by deeper study. An analysis can occur even in the absence of the "Holy Grail" of an adequate severity adjustment system.

This list of definitions is important to understand, as its elements frequently justify requests to ignore the use of information as an analytical tool altogether, or to ignore conclusions implied from analysis.

Unfortunately, health care databases are frequently characterized by one or more of these conditions. Nonetheless, their availability requires our attention and, in proper hands, reveals a wealth of usable clinical information. There is significant time and business pressure to use data that currently exists to formulate management decisions for action, pay physicians, or make actuarial predictions before the purists might pronounce the data to be "perfect."

A benefit director from one of our local major corporations told me once: "Regularly, you send a patient to the operating room with a normal white count for what you believe is an acute appendicitis and are correct. You, in fact, make life and death decisions with imperfect data on a regular basis. So, what's the problem?"

Why is moving ahead with imperfect data important?

To begin with, imperfect data is often all you have. Secondly, serious problems within the systems for financing health care reform or improvement cannot wait for an ideal world to emerge. Finally, information contained within the data is, in fact, useful and can be employed for business and clinical planning. Real time perfect data reporting is often unnecessary, and population sampling statistics, outiler analysis, and non-parametric statistics can be effectively brought to bear given non-normal distributions or small sample sizes. It is how the data is used that is important.

Understanding sources of error in health care databases is also an important prerequisite to acting on information. These include:

* Financial or clinical model problems, such as the absence of a good severity adjustment indicator. Adjustment in the sensitivity of the statistic, or vigor with which the results are used should be influenced by this real problem. Continuous model refinement must follow.

* Database problems, such as problems with information data entry by clerks or database creation by programmers. The tendency of health care "legacy systems" to use different codes for patients and physicians, making a later merger of databases difficult, is another example.

* Statistical chance. It is possible to have a bad month or week. Statistics may indicate someone is having a quality problem (high utilization, high mortality rates), when trends through time later show this is a statistical fluke.

* A statistically correct finding or area of concern. Clinical or financial significance can be demonstrated only after the first three sources of error are eliminated or adjusted. It is very important that statistics alone not be used to define guilt. Statistics at any level can only point to where a problem might be. Other methods of analysis or even chart review may be necessary to make a final decision. Often, "statistically correct" can only be estimated or inferred from imperfect data. It is absolutely necessary to have clinicians to review the results for validity, applicability, or appropriateness and to understand the impact of incorrect inferences.

The following recommendations are proposed to initially deal with problems in data imperfections.

* Develop and use approximate conclusions. "Good enough" is frequently sufficient to draw meaningful conclusions. Listings of the top 10 DRGs by total impact are unlikely to be affected by minor irregularities, sample size, etc., and can be used to focus Qi, CME, or disease state management program attention or other similar types of activities. At the practice level, doing a sampled analysis of 300 consecutive clinic visits will very likely produce a good idea of diseases treated in the practice or assay patient satisfaction levels. Recognizing that sampling techniques or data limitations are operating is important in this approach.

* Employ sample population statistics. Standard tools exist that allow the use of sample populations as an approximation to the whole. Using such tools in the HEDIS 3.0 sampling methodology is an example of how this technique is an adequate substitute for entire population polling. The statistics will convey how much "noise" can be statistically tolerated, and force intelligent use of the results.

* Recognize and adjust for sources of systematic error. If your patient referral statistics are calculated differently than "reference" definitions, then statistics may need to be simply adjusted as appropriate. Failing that, longitudinal trending of a consistently applied process might be used to determine whether your internal benchmark is changing over time.

* Information mining techniques can be used to develop proxies for indicators you want. To study cholesterol levels, for example, you may not be able to get exact results from a laboratory database. However, your claims database may allow you to isolate who did or who did not have the tests and, thus, use the results to either collect additional information, perform audits, or in other ways monitor your population for cholesterol testing. You may not know what the levels are, but if you find that only 10 percent of your population has been tested when cardiovascular disease is your number one diagnosis, than you have used imperfect data to suggest an initiative to increase cholesterol testing in at-risk populations.

* Use imperfect information. To someone with a hard core science background, this sounds heretical, but basing clinically-oriented business plans on information with some error is possible if you understand and appreciate how much imperfect data is present or might be present. You can assess how many significant financial consequences will follow if the error exceeds your estimation. Knowing that physical therapy expenses are three or four times above target, for example, may be of great use to the contracting staff or physicians, even if data down at the individual provider level may not be reliable in a statistically significant way to point out "bad utilizers. Percentile ranking of raw statistics will cause individual providers in the extremes to review their current practices, even if your confidence in the data will not permit you to take punitive or "best practice" oriented actions. Moderate confidence in a statistic may still yield enough of an idea of what is needed that action can proceed.

* Use creative strategies to manage data. For example, while it may be difficult to create a perfect "specialist profile' for gynecology when many practice patterns exist (OB/GYN, GYN only, infertility specialists, etc.), one could define a profile including other useful information. The volume of procedures done might reflect how busy or experienced a practitioner is. Calculating practice preference-independent quality statistics, such as "the probability of hysterectomy in a woman 35 to 50 years old who presents with dysfunctional uterine bleeding," may, in the aggregate, help to characterize the surgically aggressive physician from the more medically oriented one. Patient or referring physician satisfaction surveys from a reasonable number of individuals are easy to perform and can offer significant insight into other performance areas.

* Follow statistics through time. Aberrancies due to chance alone usually do not persist or get worse with repeated measurement. Those favorite Deming run charts or control charts are really effective tools here.

Not an absolute judgment system

Clinicians still tend to (rightly) have problems when imperfect data is used for important comparative purposes to profile their clinical performance for payment, public review, or compensation. It becomes particularly acute if such information is used in payment compensation algorithms. As scientists, physicians prefer exact numbers on specific patients when plans may be only able to provide population statistics. Yet, as trained analysts, applying proper statistical methods and reasoning can be compelling arguments to accept strategies in attempting to create and move forward with business activity, despite the presence of imperfect data.

As an example, outlier physicians often request "drill down" on their performance statistics until some rationalization can be generated to prove that they are excellent physicians after all. As we "peel the onion" towards more specific information, it is possible to peel all the way down to nothing, a common problem precipitated when IS staff, medical directors, and managed care organization representatives are caught in the tedious, one case at a time explanations of individual patient histories. These types of occurrences can be extremely frustrating, as individual practitioners search for rational justification to explain a clear problem, such as referral rates that are four times a provider's average, or per member per month cardiology costs that are three standard deviations from the mean for an average individual physician's panel. At the hospital level, UM and QI committees are often configured to act as defensive units, to correctly answer why a seeming" outlier occurred within their particular facility.

It is incumbent on medical managers to move individuals off the mark of attempting to explain away all data outliers and, Instead, to focus their attempts at using the information to create continuously improving business and clinical practices. It is important to get them, (and the creators of data) to move from thinking of the information as an absolute judgment system, and to focus on using the information in a learning, improvement mode. In the case just described, "even if we are not guilty, it won't happen again" is a good response. The "outlier" statistic should be used to form the nidus of the next CQI project or to encourage a "best practice" search and comparison from other providers within the database.

There is also a pressing need to move from a judgmental nature when presenting data to clinicians. Health care workers are extremely sensitive by training to the association of less than perfect performance with failure, lawsuits, and other negative consequences. It is important to identify the fact that all individuals can be expected to have some imperfections in their statistics and that they can serve as a source of continuous improvement in the learning system rather than in a judgment focused system of analysis, In truth, lack of severity weighting algorithms and data limitations can make conclusions difficult to achieve, but the use of data and information to guide action can still occur.

Moving data into effective action plans

The successful use of imperfect data requires an integrated team of information systems experts, management oriented individuals, analysts, and physicians who move data into effective action plans. This most effectively occurs (please see Figure 2) in the following five-step approach:

1. Data generation. At the local and plan level, good data collection and storage is essential for later success. This somewhat unglamourous task must have the constant attention of supervising individuals to insure that data is properly collected and stored, often by individuals who may not appreciate its later relevance and use (medical practice office clerks, computer database programmers, or other individuals without clinical or management backgrounds).

2. Information generation. Data must be converted to useful information if the next step is to proceed. Limitations must be clearly noted and spelled out (e.g., if mental health data is carved out, if certain hospitals are paid differently, if claims from the delegated facility are always late, etc.). Information presented to and from analysts must clearly indicate such limitations if confidence boundaries of the analysis can be properly passed to the individuals using the information.

3. Analysis. Multi-disciplinary views are needed to analyze health care data, but all analysis must, in the end, include overall clinical direction before completion is declared. Typically, nonclinical IS staffs have supplied reports to clinical staff or managers without clear direction or an iterative/reiterative approach. The most successful strategy for creating meaningful business plans is to have information systems and reporting staff generate cleanly defined databases that are read and manipulated by clinical people with intermediate level IS skills. An iterative/reiterative approach to report generation by clinical individuals actually operating the reporting tools themselves is highly effective in getting to information that is needed in the minimal amount of time. Providing defined databases to IS literate clinical analysts, rather than reports, assures maximization of individual efficiencies.

Data analysis must also deal with the inevitable generation of unflattering information with any one of a number of variables for any or all providers. All individuals working with the data should feel assured that data regarding their performance will be held confidential until all parties are convinced that the information and action plans are appropriate. The approach should be collegial, emphasizing a continuous clinical or business improvement atmosphere and recognizing that unflattering statistics occur with every provider in a practice or plan.

4. Business plan formation. Of the information developed, it is usually evident that some things require action. For example: A group practice has per member per month physical therapy expenses three times normal; the hospital mortality rate for pneumonia Is high compared with all hospitals; allergy referrals by an individual physician are considerably above benchmark estimates; and referrals for G1 procedures for a geographic region are rapidly increasing on a yearly basis.

Clinical staff need to evaluate whether or not the observed statistics are easily explained and understandable, whether actions can be implemented which are under the control of the providers involved, or if further analytic refinement is needed. It is also extremely important to emphasize that the statistics must have clinical analysis and confirmation before conclusions regarding action are advanced. Hasty determinations regarding which clinicians are "bad" are one of the major impediments in getting those providers to later participate in an improvement process. It should be emphasized that statistics indicating outlier status only point to where an intelligent person should begin to think and take action.

Developing an action plan should include identifying specific problems, tasks to be completed in addressing them, the start and target completion times for the tasks, and a single, accountable individual to oversee their performance. Periodic reports of the accountable staff to colleagues will insure progress, as well as provide education to the team overall. The approach to business plan formation, in turn, should include standard CQI measurement tools and strategies. A sample action plan is attached in Figure 3.

5. Operationalize a solution. Finally, placing the activity "online" needs to occur to insure the problem won't develop again, and is probably the second most unglamourous portion of this process (after data entry). It is here where most efforts at making an impact with data often fall. Policies and procedures need to be written, approved, and disseminated. Audit tools and other periodic tests of ongoing function will assure closure of the entire process. Closure of the CQI "loop" needs to show the actions fixed the problem, and will encourage celebrating the achievement and act as a catalyst for further activities.

Finally, celebrating success and retelling the process validate the usefulness of all the hard effort contributed by staff, and add to the likelihood that individuals will want to bring these projects to completion. Longitudinal improvement and performance stability are attributes of good quality and business planning and will serve as proof of a good program. While each of the steps described may seem fairly trivial, it is easy to recognize many occasions in all organizations where real achievement has failed, due to breaks in the process.

Conclusion

Recent developments in the acquisition of health care information have resulted in massive "data warehouses" compiled and maintained by information systems professionals. While these large databases have significant limitations due to historical and human reasons, a wealth of information can be isolated from their elements. Proper analysis requires a well trained, integrated team of disciplined individuals employing analytic tools to isolate information useful to clinical and financial managers in managed care organizations. Connecting this information with effective business work plans will be a success fundamental of competitive group practices and managed care organizations in the next decade, Physicians, hospitals, and managed care organizations ignore the need to do this at their peril.

FIGURE 1

IMPERFECT DATA

Name  A/K     D/K  Ref  Data A  Data Z

ABCD  55      234  3.4          200
BCDE  64.4    254               200
CDEF  54.8    456  4.2          200
DEFG  33,567  12   5.4          200
EFGH  35.6    256  2.9          200

Characteristics of Imperfect Data

* incomplete, unfinished

* holes, absent fields

* local quality deterioration

* insufficient to first desired use

* time fragmentation

* non normal distributions

* no severity adjustment (severity, complexity)

Donald E Fetterolf, MD, MBA, FACPE

FIGURE 2. OVERVIEW

Characteristics of Imperfect Data

* Data

* Information

* Analysis

* Recommendation/Action Plan

* Operations/CQI Business Production Follow-up

* Donald E. Fetterolf, MD, MBA, FACPE 

FIGURE 3

SAMPLE ACTION PLAN GENERATION

Topic     Action      Resp        Start   Target  Status

Physical  Adopt       Dr. Jones   2/1/98  5/1/98
Therapy   Guidelines  Practice 1
Referral

Physical  Low Back    Practice 2  2/1/98  5/1/98
Therapy   Review
Referral

Allergy   Review      Dr Smith,   2/1/98  5/1/98
Referral  Data        Practice 2
Product   Present     Medical
Variance  Data to     Director
          All MDs

Donald E. Fetterolf, MD, MBA, FACPE

Acknowledgment

The author would like to thank Manny Bellmore for the original formulation of the "sources of error" elements in this article.

Bibliography

Fetterolf, D. "Evaluating Medical Outcomes Statistics," Health Care Guide. Warren, Gorham, and Lamont. 1994. 3.051-3.055.

Fetterolf, D. "Evaluating Medical Outcomes Statistics," Journal of Health Care Benefits. January/February 1993, pp 46-50.

Bailar. J. and Mosteller, F. Medical Uses of Statistics Second Edition. NEJM Books. 1992.

Pfaffenberger, and Patterson. Statistical Methods. Irwin Press, 1987.

Donald E. Fetterolf, MD, MBA, is Senior Medical Officer of Alliance Ventures, Inc., a subsidiary of Highmark, Inc. He can be reached by calling 412/544-8854, via fax at 412/544-6792, or via email at [e-mail].