Herding stats: the quality measurement dilemma.

Early efforts at improving quality were undertaken in the 1970s, when the potential for improvement was realized by social scientists and the professional standards review organizations (PSROs) debuted in 1973. But by the late 1970s, the emphasis shifted to cost containment.(1)

The introduction of the prospective payment system (PPS) in 1982 and the impact of Deming's quality management principles the same year combined to rejuvenate interest in quality and, in fact, contributed to the evolution from quality maintenance to continuous upgrading.(1)

The theory of continuous quality improvment (CQI) or total quality management (TQM) evolved from the manufacturing industry. In 1987, the National Demonstration Project on Quality Improvement in Health Care showed that the CQI techniques employed by industry could be employed by the health care. This was followed by the emergence of benchmarking and critical (clinical) pathways, practice guidelines, and outcomes management.(2)

Despite all these efforts to improve quality, documentation remains a problem. According to the Inventory of External Data Demands Placed on Hospitals:

* Data and information developed for a specific purpose are often used for disparate applications and are invalid or unreliable in other situations.

* Providers and external organizations have developed unique data definition, data collection, and data coding conventions, all unique to their organizations. This means the data are not comparable or useful for decision making in another setting.

* External organizations are often unable to massage provider-supplied data to gain meaningful information. External organizations often request data and information not routinely collected by health care organization, and they request data that providers may collect but do not usually process, analyze, or report in the requested format.(3)

Nigel Roberts, MD, Vice President, Nationwide Insurance Enterprise, Columbus, Ohio, asserts that the biggest change in the health care system occurred in March 1986 when the New York Times first published HCFA morbidity and mortality results. "The genie was out of the bottle," says Roberts. "You could never go back to a point where there weren't published results. Providers scrambled to explain medical outcomes. Once you have outcomes, you add accountability to the equation." I think what you are seeing is exactly the movement toward outcomess and accountability on those outcomes rather than simply looking at structure and process," Roberts says.

Roberts says the good news is that there are plenty of bright people ready to offer solutions to improve our health care system. The bad news, he says, is that we've got plenty of reasons to improve the health care system.

Quality expert Paul Batalden, MD, who now teaches the subject to students in master's and PhD programs at Dartmouth College, says of the three characteristics of our environment that are impeding quality, measurement looms largest. He believes "we live in a festival of measurement." He says trying to create a measure that will work for all parties and purposes is like trying to find the universal word. "The universal number is no more or less possible than the universal word to describe a phenomenon."

"And point number two is that we have a payment system from hell." Batalden asserts that most payment systems have fundamental contradictions that "freeze the creative energy that could be applied toward change."

The third impeding factor Batalden points to is health care professionals who are hobbled in their attempts to improve quality by a basic lack of knowledge of how to achieve it. "So you have well-meaning professionals and professional leaders struggling to be responsive, to make changes, in the midst of this festival of measurement, a payment system from hell, and no prior preparation," he sums up.

David B. Nash, MD, MBA, Director, Health Policy & Clinical Outcomes, Thomas Jefferson University Hospital, Philadelphia, also paints a bleak picture. "There is an industry out there producing guidelines," he says. "There is no evidence that they make a difference, and there is mounting evidence that they are very hard to implement and to use."

Nash is currently involved in a $2 million-a-year research project to develop ways to improve clinical guidelines and other quality tools so that documentation of value can be proven. "The question is, 'If I follow the guidelines, will things turn out better? Or will things turn out, hopefully, as expected?" asks Nash. "That's the missing link at the moment."

Cary Sennett, MD, PhD, the first physician to hold the position of Vice President for Performance Measure at the National Committee for Quality Assurance (NCQA), also believes quality measurement is faulty. "While I think there's been a lot of theoretical reasons to believe that guidelines have value, and a lot of people anecdotally recognize the value of guidelines, until there's a way to systematically measure the impact or performance of health care organizations, there won't be a way to document that using the guideline has improved the outcomes or improved the effectiveness of care," Sennett says. "What's been missing is the yardstick, the measurement tool that lets us recognize that an intervention, such as the implementation of a guideline, has led to change."

Donald Berwick, MD, who heads the Institute for Healthcare Improvement, Boston, and publishes the Eye on Improvement newsletter, reports three trends in quality. "There is more measurement of quality, much more," he notes. "There are formal report cards showing cost and features of quality, patient satisfaction, length of stay; it's being demanded by purchasers."

Also significant is the trend toward purchasers' making demands based on data. "The jury is still out on how many organization are actually improving quality," Berwick points out. "There's a very steady subset of organizations that have been making improvements for the past four to seven years; the best of them improve, but I don't know if, as a whole, the industry is improving."

Finally, Berwick says there's a creeping awareness that cost reduction and quality can be linked strategically, and Consultant Martin Merry, MD, agrees. "There is a growing realization that lower cost and improved quality are on the same side of the coin," Merry says. "I'm seeing a lot of comparative quality data, and the old excuse, 'We have sicker patients,' is no longer good enough."

But Michael Massanari, Medical Director, Professional Practice Review Group, Henry Ford Health System, Detroit, worries about physicians' ability to balance cost and improved quality. "There is no doubt we are operating very inefficiently," says Massanari, "but, on the other hand, there is a strong emphasis on cost issues. Physicians are beginning to equate reduction in cost and utilization with quality, and I think we need to maintain a healthy balance between quality and cost."

Massanari explains that quality can be defined as "optimizing benefits while minimizing risks, without factoring in costs at all." He says when you do factor in costs, you're looking at values. "What we want to do is maximize value, which means we maximize the amount of quality for the amount of dollars that we spend on care," he says. "My concern is that people equate cost containment with quality and fail to take the broader view of value."

Massanari says that scientific evidence for the efficacy of quality improvement is still wanting. Despite today's advances, he maintains the best scientific evidence came from the infection control programs of the late 1960s and 1970s that were mandated by the Joint Commission. "Proponents of CQI and TQM often criticize these programs, linking them to traditional QA programs," he says. "However, hospital epidemiologists utilized measurement, scientific method, feeding back of information to physicians (particularly wound infection rates to surgeons), and there is evidence in the literature that these interventions changed their behavior and improved the quality of care."

Grading Report Cards

Report cards have only been around for a few years, but they have generated a hailstorm of controversy. Like it or not, however, most of the experts predict report cards will be around awhile longer. And most believe that, with improvements, they could become valuable indicators of quality.

Cary Sennett says that, while today's report cards are primitive and are used primarily by corporate purchasers, tomorrow's report cards will offer more information and could be used by public purchasers or regulators to create consumer awareness.

David Nash says that, despite their faults, report cards are one of the best tools now available to measure quality. "Report cards will improve; they'll have more sensitivity and specificity, and they'll get wider distribution. And the public will have access to increased amounts of information," says Nash. "Whether they'll actually use this information in the short term, I don't know. But I do believe that, in the long term, report card information will be critically important."

Paul Batalden is less enthusiastic. He says report cards are part of the festival of measurements. "Not only are they new, but they don't address any of the things you would use to measure quality," he scoffs. "We think they're so good in health care that we're investing millions of dollars in them. It's just that they really doesn't seem to meet the purpose for which people say we need them. Other businesses would call that waste."

Information Systems: An Important Quality Component

Many clinicians and physician leaders are becoming knowledgeable in information systems and proficient in informatics so they can have input in the design of these systems and so they know how to capture and analyze data for outcomes. "There is a need for physicians who have a good grasp of informatics by way of what's perceivable in the equipment and what software design is all about for the development of software programs," says John Pufahl, EdD, President of Avalon Health Group, a New York City-based physician executive recruiting firm. "The average medical director does not have to be an expert, doesn't have to be able to sit down and write code and design architecture, although, upon occasion, it's quite helpful. But he or she should be conversive and have the ability to understand how the analysis works."

Michael Massanari laments that health care organizations develop their information systems to support financial programs, not clinical care. "We have a long way to go to upgrade our information systems to support clinical care and, in turn, to support measurement," says Massanari. "The few exceptions are LDS Hospital in Salt Lake City, Columbia in New York City, Beth Israel in Boston, and some other examples around the country. But, by and large, even though the technology exists, we in medicine are well behind the state-of-the-art information systems."

While many of the experts say information systems haven't reached a point technologically where they can accurately collect data, Paul Batalden says that's a cop-out. In fact, he offers a scenario to back it up.

"The provider says to the information systems person, 'I don't have the information I need in order to make the changes I need to make.' "The information systems person says, 'Tell me what you need.' She, eager to be perceived as cooperative, does her best articulating what she perceives her needs to be. He takes copious notes and says, 'My goodness. This is quite a different kind of information system you're going to need. I'm going to have to go back and do some figuring on this.' Later, he returns and says, 'It turns out we're going to need not only new software and new report formats, but we're also going to need new hardware. This is a major investment.' "She says, 'We're going to have to take this to the Board.'

"Justifications are prepared. The Board sighs deeply and says, 'Well, we've got to do what we've got to do,' and then makes a decision to invest in a huge new information infrastructure. Time elapses. She's betting this new information system will meet her needs.

"The new system arrives. But it doesn't work. It doesn't meet her needs. One of her partners says, 'You know, I've been through this before. That's what's been going on in the data logs between information services people and health providers.' So, the refrain is almost boring, and it's because the fundamentals are missing."

Batalden maintains that it's the job of the information services professional, not the provider, to understand the customers' needs. The provider can only convey perceived needs. "But we have been willing participants in this charade," he says. "It's time to call it. Move beyond it."

HEDIS 

Cary Sennett believes HEDIS is the most successful example of performance measures, the technology that's needed to support a report care or to support comparisons of health care performance and objective measures. Of course, NCQA, the organization that pioneered this set of guidelines, is also his employer. Despite this bias, Sennett is realistic about HEDIS' shortcomings. "HEDIS is not a full-formed or finished product," he admits. "We need to build on what we have, expand it, enhance it, and work honestly to continuously improve it," he says. "It's also been an important driver for improvements in information systems."

He adds that HEDIS has forced a tremendous amount of attention on the quality of the data collected, something that wasn't considered important in the past. But there are still weaknesses that must be addressed. "We need to extend HEDIS to other populations, other purchasers, make it useful to other purchasers, not just the commercial purchasers," he says. "So we are working in exploring the possibilities of extending or adapting HEDIS to the Medicaid population and conceivably to the Medicare population."

Sennett says HEDIS currently focuses on preventive care and that NCQA, under his direction, is working to expand that focus. "We need to think about how HEDIS is going to wear in a dynamic world where technology is changing, practices are changing, the tools for improvement are evolving, and so on; that's what the next generation of HEDIS is about."

Until HEDIS 3.0 is released, Sennett says, there is a need to continue to improve the process. In fact, a set of revisions submitted by HEDIS users has recently been released as HEDIS 2.5. "If the production process truly is going to be standardized, there's a need for us to focus not only on specifications but also on whether people are delivering statistics according those specifications," he says. "There's really a need to audit this process or in some other way document that people are using the standards we laid out."

David Nash agrees that HEDIS needs some work. He views it as a "short-term, report card-like measure." In its current form, Nash says, little can be learned, but he has faith that it will be a valuable tool to make comparisons.

The Joint Commission's Role

According to Frederick B. Fishburn, MD, a surveyor with the Joint Commission, quality initiatives equate to "a race with no finish line." Despite the grim prognosis the experts give the current state of quality, Fishburn says that, in his 40 to 50 visits to hospitals each year, he's seen some remarkable changes. "There's a much better dialogue now than when I was on the other side of the coin, when I was a chief of staff, when I was a hospital commander. I think the process has really improved," says Fishburn.

Fishburn joined JCAHO in 1983 after he retired from the U.S. Air Force. "I joined the Joint Commission because I thought it was changing the vanguard in medicine," he says. "The philosophy of the Joint Commission is that there should be one level of care for the little and the big guys. Are you doing the right things? Are you doing them well? Are you continuously improving? I think there's a great story to tell about how American hospitals are doing."

As for those who question the value of being surveyed by the Joint Commission, and many have of late, Fishburn has an answer. "I think those of us who take pride in our profession are pleased when our peers come in and tell us we're doing a great job," Fishburn says. "We try to make our visits user-friendly and comfortable. The Joint Commission is dedicated to the idea that hospitals are customers. If they think we give them a hard time, they can tell us by telephone. We give them forms to evaluate us. If we can't perform our oversight function, maybe the organization doesn't need to exist."

Fishburn believes the tools necessary for improvement are available, and he believes that, as quality changes, so will the survey process. He foresees surveyors using laptop computers and databases that will allow visits to be more consultative in nature. He also predicts more regionalization of surveyors and hospitals and more surveyors to work in the regions. "The goal is to make the process more effective."

New Collaborations

During his eight years at Quality Health-care Resources, Inc., a new not-for-profit consulting subsidiary of the Joint Commission that works with health care organizations for presurveys and continuous quality improvement, Edgar Blount, MD, has visited more than 300 hospitals across the country. One of the newest lessons he's learned is how to cure the "four peaks syndrome." Blount describes it as "the administration on one peak, the board on another peak, the doctors on another peak, and the nurses on another peak. Two of the peaks are ignoring each other, and the other two peaks are throwing rocks at each other. The whole movement toward trying to improve quality is really trying to get all those people off the peaks and into the same boat, going down the same river, making the appropriate decisions for providing high-quality health care. This collaborative aspect is something relatively new for quality. I think we've always tried to do collaborate, but I think we are finally learning to do it. The new Joint Commission standards for leadership and improving performance actually help to provide a structure to achieve a more horizontal organization."

The Role of the Physician Executive in Quality

Paul Batalden believes physician executives "need the knowledge and the skills that will be helpful to them in understanding the enterprise as a system. They need to understand how to interpret and look for the variation that is resident in that system, day in and day out, and that is now causing cost and waste and rework," he says. "They need to understand the relationship of the policies and procedures that exist within their organizations and those things that permit people to take pride and joy in their work. They need to know what knowledge and skills are necesary to foster the creation of a learning environment for work, because change without learning is not fun for health professionals."

Martin Merry puts is more succinctly: "At some point, we have to become agitators so quality won't diminish," he says. "We have to throw out the challenge to keep moving and looking for better ways to serve the patient."

References

(1.)Stiles, R., and Mick, S. "Classifying Quality Initiatives: A Conceptual Paradigm for Literature Review and Policy Analysis." Hospital & Health Services Administration 39(3):309-26, Fall 1994.

(2.)Nash, D., and Johnson, N. "The Elements of Medical Quality Management." In New Leadership in Health Care Management: The Physician Executive. Tampa, Fla.: American College of Physician Executives, 1994, pp. 229-37.

(3.)Longo. D., and others. Inventory of External Data Demands Placed on Hospitals. Chicago, Ill.: Hospital Research and Educational Trust, 1990.