Physician report cards: now and in the future. (Value-Based Health Care).

SINCE THE INVENTION OF THE "ORGANIZED MEDICAL staff" in 1919, (1) hospital/medical center leaders have sought to provide evidence that practitioners provide dependable medical care.

Such efforts are as important today as they ever were. That's because while we tend to focus on the task of effectively integrating all kinds of health care services, the public's primary interest is that people receive dependable care on the scariest days of their covered lives. Besides, managed care contractors now want more than fiscal efficiency. They also insist on data confirming dependable performance, including physician report cards. (2, 3, 4, And, the American Medical Association has now offered to provide physician accreditation. (6)

Qualifications versus performance

Concern with physician qualifications ("credentialing") is different than concern with ongoing performance. In the past 20 years, credentialing methods have evolved from a pro forma inquiry into a sophisticated process. But methods for evaluating a single physician's performance have not kept pace. One reason is that absolute requirements, such as "12 audit studies a year" discouraged, indeed stifled, experimentation with methods.

Another block was our insistence on using the term "quality" to refer to this activity, then getting stuck before we got started because we didn't know how to define it. A third reason for dragging our feet is fear of legal complications if we admit that one of our doctors needs to improve either clinical judgment, technical skills, or attitude.

So today, we find ourselves ill-prepared to shake off the paradigm of developing physician performance information just to comply with Joint Commission standards and relevant State and Federal statutes. While there is a genuine need for physician performance reports that are as sophisticated as credentialing and financial reports to the organization's strategy for successful economic competition, we are stuck with six basic methods of reporting physician performance, all of which are seriously flawed.

1. Peer review

This ancient exercise consists of a physician peer review committee sitting around a table, thumbing through patient records, looking only for physician mistakes. The well-known result is either a report that "There are no problems this month," or a request that the medical executive committee and board authorize preparation of "charges" and convene a "hearing" to correct the problem physician. (7)

Today, peer review should be used only when it is necessary to refer to a strict legal procedure described in relevant State statutes and in the combination of board and medical staff bylaws.

2. Essay-style reports without accompanying data

These reports provide a description of physician reports without any supporting data, such as "All patient care this month was appropriate."

3. Raw data without conclusions

Statistics, such as the number of hospital admissions, clinical consults requested, and transfers to another hospital are useful for some purposes. But they do not address whether or not physician clinical decision-making and technical skills (surgery, invasive procedures) have been dependable. For example, why was this patient transferred to another hospital instead of being discharged home needs to also be addressed.

4. Statistical reports that do not isolate physician factors

If the purpose is to compare Hospital A and B, then benchmarking--using gross data, such as post-operative infection rates--might be helpful. If you are the patient, you'd rather be In Hospital A with a post-operative infection rate of 3 percent than in Hospital B with a post-op infection rate of 9 percent (if that data Is reliable and all other factors are equal). Reaching that conclusion does not require sorting out all the reasons for the higher post-op infection rate in Hospital B.

But one cannot draw valid conclusions about a physician's performance unless the evaluation method includes the step of isolating physician factors that contributed (or did not contribute) to patient care results in statistical reports.

5. Outcome analysis

Some still believe that dependability of physician performance can be deduced from data about end results of patient care, such as mortality rates. This approach holds promise, but is not yet reliable and will probably prove to be just one piece of information needed to adequately evaluate a physician's performance.

One example of this approach was the attempt by the Health Care Financing Administration to compare arbitrarily determined 'expected deaths" against observed "actual deaths" and, by statistical manipulation, to calculate an arbitrary number labeled 'unnecessary deaths." Because of the many limitations in this methodology, HCFA has abandoned that effort. (8)

Another example is the attempt by New York State to develop report cards on cardiac surgeons performing Coronary Artery Bypass Grafts (CABGs). New York claimed a subsequent reduction in mortality following CABG due to the physician report card exercise. However, objective analysts point out serious problems in the exercise and in the state's conclusions. (5) One factor in the mortality rate reduction might have been that some surgeons, concerned about the report cards, elected not to operate on sick patients who most needed the procedure. ("Number of patients in which an operation was considered but decided against" would be a very useful performance indicator, but, so far, is not included in physician performance reports).

Also, since the report cards were weighted according to risk, the incidence of reported risk factors increased in the state of New York to a miraculous degree. In one hospital, for example, reported prevalence of chronic obstructive pulmonary disease in this patient population increased from 1.8 to 52.9 percent. And, in another, the incidence of unstable angina increased from 1.9 to 20.8 percent. In addition, "there may even have been a trend toward referring such cases out of state" (5)

6.A report of physician practice variation from established guidelines or clinical protocols

This aggregate variation from guidelines statistic can be quite misleading. That's because there are many reasons that a physician's care plan for a specific patient may differ from practice guidelines. The reasons vary all the way from arrogant disregard for the guidelines, to adjustments made necessary by specific reactions of the presenting patient, to disease processes and chosen treatments.

Here is an example which I know to be true, because I was the patient. A 48-year-old man was diagnosed as having a giant cell tumor of the sacrum, invading nerve roots and threatening vital functions. Conventional wisdom was that this particular tumor cell type was resistant to radiation therapy. The patient's specialist elected to try radiation therapy anyway, since immediate definitive surgery might have left the patient crippled and incontinent. The radiation therapy worked. The tumor was eradicated, and the patient's vital functions were preserved. Shall we really simply characterize this physician's good judgment as "deviation from practice guidelines," and tabulate this case as a negative on the physician's performance report?

The big fix

In the language of Continuous Quality Improvement proponents, the step of defining causative factors (answering, "why?") is called cause and effect analysis. In this parlance, the format of truly useful, substantive, valid, and fair physician performance reports would be cause and effect-adjusted physician performance data.

For example, go back to the "number and percent of variations from practice guidelines and clinical protocols" statistic (number 6). Start there, but take a different road. Instead of "analyzing the data" by mathematical manipulation of statistics, explore cause and effect analysis. Why did this particular physician in this specific instance depart from this practice guideline? The resulting physician performance report can then summarize conclusions about whether or not these reasons represent dependable patient care or poor clinical judgment. Here is a simple example of how a resulting report might look:

Physician Performance Report First Quarter, 1997


Physician                                       #23

Total admissions and clinical consultations on   29
 hospitalized patients

DP (Dependable Practice)                         17

RC (Reasonable clinical controversy)             11

DR (Discussion Required)                          1

PR (Sent to Peer Review)                          0

A plan for proceeding

So, what should the Vice President of Medical Affairs do?

Let's face it. This probably isn't your primary area of expertise. The CEO expects you to spend most of your time on insurance issues, entrepreneurial innovations, managed care contracting, and networking with other health care agencies and the community. So, you aren't expected to become an expert on the technical details necessary to put together a physician performance report.

However, you must provide leadership and support the medical staff and PAMI office (Performance Assessment, Maintenance, and Improvement). And you must provide orientation, guidance, and support to physician leaders, such as chairs or directors of clinical departments.

Here's a plan for proceeding, which can be modified to suit the reader's specific needs.

1. Provide training

Be sure the medical staff and PAMI directors/coordinators have the training, experience, and skills that match this task. In one sophisticated Midwestern hospital, the person depended on to provide useful information to physician leaders was a highly-trained and brilliant PhD with work experience in developing epidemiological statistics. Unfortunately, the physician leaders found the raw data provided to them by this individual unnecessarily complex, and relatively useless for the specific purpose of evaluating physician performance.

Expect even experienced personnel to need some coaching and retraining. Remember, these individuals will be steeped in one or more of the six flawed methodologies.

2. Review this article with staff

Make several photocopies of this article. Distribute it to the medical and PAMI staff. Assign them the task of reading the article and preparing to discuss it, then give them time to accomplish this during a regular work day.

3. Discuss alternative methodologies

Facilitate a working meeting and discuss the concepts in this article and alternative methodologies. Decide on next steps in experimenting with methodology or decide to simply continue whichever of the six current methods are in use in your medical center. Thus endeth the value of sitting around and discussing physician performance report methodology in meetings. Staff should now go to work to accomplish the agreed-upon next steps.

4. Compile a glossary of terms

Help the staff to compile a glossary of terms relevant to developing physician performance reports. For example:

Quality

In the context of PAMI activities, "quality" means dependable performance from the viewpoint of patients and their family members, (Note: This does not mean that patients and their families make judgments about whether performance is dependable, at least in the context of knowing when reasonable clinical knowledge and skills are being appropriately applied. It does mean, however, that when physician analysts, panels, and/or committees state conclusions about a physician's performance, these conclusions are from a "Pretend You Are The Patient" point of view.)

Peer Review

This term is never used except to refer, as necessary, to the specific method/activity described in relevant statutes and in the combination of medical staff and board bylaws.

Physician Analyst

This term means that a member of the medical staff with relevant clinical expertise, designated by a clinical department chair, accomplishes the intial analysis of physician performance data, observations, and findings, working with PAMI staff. The physician analyst has no authority, except as delegated by the department chair who appointed the him or her.

Add working definitions to the glossary as they occur to you and others while developing physician performance reports.

5. Post signs

Provide the following signs (in large print) to the medical staff and PAMI office to post as constant reminders:

WHEN THE PURPOSE IS DEVELOPING PHYSICIAN PERFORMANCE REPORTS, CLINICAL SIGNIFICANCE IS AS IMPORTANT AS AND SOMETIMES MORE IMPORTANT THAN STATISTICAL SIGNIFICANCE. WE ARE NOT DOING RESEARCH, AND WE ARE NOT ESTABLISHING TREATMENT PROTOCOLS OR PRACTICE GUIDELINES. RATHER, WE ARE EVALUATING PERFORMANCE. THIS CAN BE DONE WITH FEWER INDICATORS THAN WE MAY BELIEVE

6. Create a statement of purpose

Frame a statement of purpose for developing and using physician performance information, approved by the medical executive panel. It should be provided to the board for information, and used by the medical and PAMI office staff as an implementation guideline.

Here's an example of a physician performance report statement of purpose:

The purpose of developing and using data, observations, and findings reflecting physician performance is to accumulate a positive physician performance profile for each practitioner on this medical staff confirming dependable performance where that is the case and helping individuals improve performance if or when that is indicated. This activity is intended to be of simultaneous benefit to each physician on the medical staff to patients and their family members, to this medical center, and to the integrated delivery system of which this medical center is a part.

7. Develop a demonstration physician performance report

Put together a demonstration physician performance report and provide guidance to staff as they develop a practice report featuring cause and effect-adjusted performance statistics.

This exercise begins with collecting data using a small set of performance indicators. The reminder that this is not research will help staff keep the list of indicators short, so that all data produced can be addressed by physician analysts. And emphasizing that this activity is different from establishing clinical guidelines and practice protocols will help avoid the traditional error of taking too absolute an approach in dealing with this information.

Help staff identify and orient some unofficial physician analysts. Stress that this is just a practice exercise with no official standing and no impact on any physician's record. Help the medical and PAMI office staff work one-on-one with these practice physician analysts to complete this exercise by providing a sample physician performance report with the following categories: Dependable practice (DP), reasonable clinical controversy (RC), discussion required (DR), and sent to peer review (PR).

8. Conduct an orientation session

Conduct a special orientation session for medical executive committee members, department chairs, other key physician leaders, nursing leadership, and board members. Suggest that receiving and discussing summary physician performance reports should become a fixture on the standard meeting agendas of clinical departments, the executive staff, and the board. Collecting and using information about physician performance is a daily, ongoing activity, but quarterly reports should be sufficient, except when specific issues arise which must be addressed in a timely manner. Ask this small working group to suggest needed improvements in this method.

9. Assure support

Finally, the VPMA and CEO must assure the support of the medical executive panel and board of directors for truly meaningful physician performance reports. This is best done by reminding executives and board members that if the current profit-taking health care model is ever replaced by a government-controlled single payer model, it may be because of public concern about dependable performance in managed care.

References

(1.) The Minimum Standard for Hospitals. American College of Surgeons. Chicago. 1919.

(2.) The Atlanta Group. America's Health Care: The Big Squeeze. page 3. The Governance Institute. La Jolla. CA. 1996.

(3.) Merry. Martin. MD. The Shifting Quality Focus: Implications for Accreditation and Regulation, pages 5 and 8. The Medical Leadership Forum. La Jolla, CA. 1996.

(4.) Kassirer. J.P. The use and abuse of practice profiles. NEJM 330:634-636. 1994.

(5.) Jesse Green, PhD and Neil Wintteld, PhD. Report Cards on Cardiac Surgeons: Assessing New York State's Approach. NEJM 332: 18, 1229- 1232. May 4. 1995.

(6.) Press release, American Medical Association. November 18, 1997.

(7.) Thompson, Richard E., MD. The Medical Staff Leader's Practical Guidebook. 2nd Edition. Chapter 6. page 43. Opus. Marblehead. MA. 1996.

(8.) Krakauer, H., Bailey, R.C., Skellan, K.J. et al. Evaluation of the HCFA model for the analysis of mortality following hospitalization. Health Services. 27:317-335. 1992.

Richard E. Thompson, MD, is the author of So You've Been Integrated. Now What: Opportunities for Physicians Practicing in Managed Care Settings (ACPE. Tampa, 1996), and of The Medical Staff Leader's Practical Guidebook, 2nd Edition (Opus, Marblehead. MA, 1996). He is a member of the Medical Staff Standards Work Group of the Hospital Accreditation Program of the Joint Commission on Accreditation of Health Care Organizations. The views in this article are not necessarily those of the Joint Commission. He can be reached by calling 813/784-1939, via fax at 813/789-4695, or via email at [e-mail].