Grading Systems for Retail Food Facilities: A Risk-Based Analysis.

Abstract Grading systems for retail food facilities have become a heated topic of debate among health professionals. Perceptions may vary with the individual players, which include environmental health agencies, retail food. operators, and the general public. The underlying purpose of a grading system can range from enhancing legal compliance to establishing incentives for organizational change, to communicating risks with the public. Whether the trigger event for a grading policy is local news coverage of food safety issues or a deliberate vision of policy makers, the effectiveness of the system should be evaluated through a policy analysis that seeks to understand individual perceptions and the underlying purpose of a grading system.

If the reliability and interpretation of risk communication is not a shared value among all the players, the effectiveness of a program could be undermined. The authors believe, however, that the current controversy presents opportunities as well as challenges.

Introduction

The evaluation of food and food facilities has a long history [1]. As we enter a new millennium, there is little doubt that food safety has been a major success story. Nevertheless, the media, activists, and legislators have renewed some old controversies (both in the United States and abroad) in the form of questions about grading systems [2-7]. A recent article in the Journal of Environmental Health clearly identified the challenges we face and issued a call for contributions to this debate [8]. Make no mistake about it--food safety is indeed a revitalized area for study and action [9].

The timing, however, creates a new context for the issue: We live in an era of risk labeling [10]. We can, for example, find risk labels for food additives such as saccharin, aspartame, and sulfites. Whether the labels appear on diet soda or wine bottles, consumers are often overloaded with risk information.

The need for a grading system has become a matter of public debate as local jurisdictions strive for the highest level of service with limited funds. Unfortunately questions about the effectiveness of grading systems often get lost in the debate. Policy advocacy without policy analysis often results in decisions based on emotion rather than fact. A policy analysis of grading systems is paramount if we are to determine whether the goals and objectives actually can be accomplished. Equally important is the education of the public about the reliability of these approaches.

These are not easy tasks. The political interests of various stakeholders present a profound challenge to the field. Fortunately, there is encouraging news for professionals faced with this challenge. The grading of retail food facilities has been identified as a form of risk labeling, and the literature on risk labeling is extensive [11,12]. Through this literature, we can better understand the limitations as well as the promises of grading systems. While the authors doubt that this analysis will, by itself, settle public controversies or identify the most appropriate policy they nevertheless hope that the incorporation of this information will elevate the level of the debate.

This paper will proceed by addressing questions about grading systems: questions that relate to purpose, differing perceptions, and measures of effectiveness. After reviewing these issues, the authors will offer general recommendations. The theme of this review is that the current controversy in various jurisdictions presents opportunities as well as challenges.

Purpose

Why Are Grading Systems Initiated?

An environmental health agency may initiate grading systems as a result of extensive policy formulation procedures, but also may initiate such systems following "trigger events" (events immediately affecting public opinion) such as television or newspaper coverage of poor food sanitation practices. The public outcry from trigger events can be magnified by activist groups, and this outcry can mandate significant changes in policy or early implementation. In addition, changes in leadership, whether from within or without an environmental health agency, may stimulate such change. In any case, one of our biggest concerns as environmental health professionals is that policy advocacy may occur without policy analysis.

Of course, grading systems may be initiated for many other reasons after more deliberate consideration. Any policy or law that lacks sufficient sanctions, for example, is often difficult to enforce. Thus, an environmental health department may decide to institute a grading system as an additional tool to compel abatement. Grading systems also may be initiated by an underlying interest in communicating risks to the public. Regardless of the trigger event or stimulus, however, we need to consider a more fundamental question that follows.

Why Do We Need a Grading System?

For all living organisms, the ability to recognize harmful environmental conditions has always been necessary to survival. The ability to codify and learn from experiences is basic. In addition, humans can alter their environment as well as respond to it. Thus, if members of the public understand the risks, they can and will make their own choices [13].

Unfortunately, modern hazards are not always easy to recognize and evaluate. Members of the public must depend on communication of risks from health officials to aid their decision making [14]. The fundamental problem is to communicate these risks to the public in a way that is understood,

The assumption that the public will use information about risks is well grounded in microeconomic studies applied to purchases of food and clothing [10]. So, if a grading system is intended to communicate risk, we can reasonably assume that rational human beings will try to include these risks in their decision making. Consistent criteria must, however, justify any letter grade, and those criteria should be understood by all parties.

Are Grades Based on Consistent Criteria?

For the sake of discussion, assume a given restaurant receives a "C" rating. Is that grade the result of temperature violations conducive to serious health risk, or is it the result of numerous "low-risk" violations? For a grading system to communicate risk effectively, the public and the environmental health professional must understand the significance of the grade. The public's understanding of risk is often constrained by limitations in the volume and quantity of information they receive. Their basic conceptualization of risk, however, is often much richer than that of the experts and reflects legitimate concerns that are typically omitted from risk communication. Each side, both the environmental health professional and the public, must respect the insights and intelligence of the other [14]. This synergistic relationship can be enhanced if members of the public understand the true meaning of a letter grade.

All of which begs the question: Is it the fundamental purpose of a grading system to inform the public about specific degrees of risk? Is an environmental health program even considering communication of risk with the use of a grading system? The authors can think of at least three different reasons for the use of grading systems:

1. to communicate risk to the public,

2. to improve legal compliance by food facilities, and

3. to establish incentive for organizational change.

If the public responds to the letter grade as a warning label, the posting of the grade could help change the restaurant's organizational structure. (This assumes the public will be selective in excluding lower-rated restaurants.) The degree of organizational change could range from an increased commitment to existing regulations to a massive reorganization of personnel. Any letter grade below "A" could act as an agent of change. When individuals in an organization understand the need for organizational change by recognizing the prospective payoffs, change may more easily occur, ultimately leading to improved food safety. The corporate culture of the public establishment focuses more on improving sanitation. As a result, the organization may become more receptive to training programs for food handlers, continuing-education programs, or even hazard analysi and critical control point (HACCP approaches. Resistance to change may b overcome by training that keeps restaurant employees current in the field, or by e ncouraging employee involvement.

There is a fundamental problem, however, in this attempt to justify grading criteria. The preceding argument is based on a series o assumptions, and assumptions may vary with individual perspectives.

What Can We Learn from Different Individual Perspectives?

From the perspective of the environmental health professional, the primary purpose of the grading system seems to be to facilitate restaurant compliance with local codes and ordinances. If the grade affects the degree to which an establishment is frequented, it provides economic incentive for the establishment to meet requirements for an A rating. It should be emphasized that whether there is significant public health risk from a C restaurant or not, these economic forces should help compel abatement. Environmental health professionals are often burdened by extra demands on time and effort stemming from establishments with a collection of minor violations that may represent minimal public health risk. Grading systems provide compliance incentive for these organizations with a minimum commitment of agency resources, and it is not necessary from this standpoint that the grades correlate with risk. To be sure, environmental health professionals are fundamentally concerned with managing risks, but there are a var iety of tools beyond grading systems to achieve this goal.

From the standpoint of the retail food facility, a grade of A may be valued as a way of increasing business. The grade would indicate compliance and limited risk. Such an eating establishment may proudly display its accomplishments as a seal of approval and as validation of its commitment to food safety.

The public standpoint is not nearly as clear. Is the purpose of a grading system to inform the public of risk? Or does the public use it as a comparison tool with no real quantification of risk? What are the differences among A, B, and C restaurants in the eyes of the public? It is especially crucial to know if the perceptions of the environmental health professional, the public, and the epidemiological research are in line.

Given the existence of differing perspectives, the authors believe the following further questions need to be addressed:

1. How are grading systems perceived by the public, the media, and environmental health professionals?

2. What are the key differences and similarities among these groups?

3. Are the criteria consistent among professionals?

4. Is the epidemiological evidence adequate to support the perception that there is an increased risk of food-related illness in establishments with lower grades?

5. Does the system compel compliance?

Perceptions

Do Consumers Perceive a Letter Grade to Be a Form of Risk Communication?

A letter grade for a food facility could be perceived as a measure of risk, particularly if environmental health professionals offer it as such. But should it be? To be rational in taking risk into account when they make choices, individuals must fulfill at least four conditions:

1. they must know the risk exists,

2. they must know enough about the risk to evaluate the costs and benefits,

3. they must understand the information presented to them, and

4. they must act upon the information.

A single letter grade does little to inform consumers about the nature of the risk or about costs and benefits. Even when additional information is provided, it is not clear whether consumers understand this information. Well-intended policies may be ineffective if the public lacks a thorough under-standing of the risks.

All of these limitations, however, point to some important opportunities for our profession. Environmental health specialists have always needed to understand how members of the public perceive and understand the information presented to them. The question can and should be evaluated. By understanding the limitations of grading, we can recommend accompanying policies to strengthen public understanding.

How Does the Public Perceive a Single Letter Grade?

If the public is interested and has full confidence in the information being presented, the critical question about grading systems is the adequacy of a single grade. A single letter grade provides minimal information.

The literature is not encouraging on this question. On the one hand, if people cannot understand a warning, a majority will assume the situation is safe [18]. On the other hand, when too little information is provided to downplay a possible risk, the opposite effect may occur [19]. In either case, a single letter grade may alter perceptions for the wrong reasons.

It is unfair and inaccurate to portray the role of environmental health professionals as simply assigning a single letter grade. By emphasizing the limitations of single letter grades, our profession can point to the wealth of accompanying information that is already available.

How Do Public Perceptions Compare with Professional Perceptions?

An evaluation of the way the public perceives risk should be an integral part of a grading program. This is especially true when government agencies do not have adequate credibility among consumers. For example, are public perceptions of the grades identical to the perceptions of environmental health professionals? If not, are the criteria used to grade a restaurant understood by the public? How consistent are perceptions among the public? Finally, how does this consistency compare with that among environmental health professionals? These questions are made even more difficult when the risk communication is limited to a single letter grade.

Are There Cultural Differences?

There may be cultural differences in the way people interpret risk labels. For even the seemingly simple evaluation provided by a letter grade, there is no reason to expect perfectly consistent interpretations among subpopulations. Therefore, such questions should be tested for the target population. Of course, "evaluate the message" is a tenet of risk communication. However, with current and future changes in U.S. demographics, it becomes crucial to question the impacts of existing risk messages on diverse populations.

Do Consumers Read Risk Labels?

Risk labels have many other known limitations. Probably the biggest limitation is that most people do not read labels [10]. Lengthy risk messages are more likely to be ignored. By reducing risk information to labels, however, we eliminate a wealth of expert judgment; worse, we present it to a public that has little training in how to interpret the information [15]. Government needs to establish a more consistent framework for evaluating and communicating risks [16]. For example, simply adding more risk labels may increase the confusion of consumers [17].

While it is critical to evaluate the message conveyed by single letter grades, such evaluation still only addresses a subset of a larger question. Ultimately, we are concerned with the effectiveness of grading systems.

Effectiveness

What Are the Significant Variables in a Grading System?

To assess the effectiveness of a grading system, we need to consider the major variables that affect a grade. There are, of course, the basic environmental health variables reflected in the scoring system. These include principles of food safety such as time-temperature relationships, cross-contamination variables, equipment, vectors, food handling, and so on.

But many other variables also may influence grades. For example, if two professionals evaluate the same restaurant at the same time (a procedure used by many local agencies for training purposes and consistency checks), are their grades the same? One inspector may rate a restaurant an A, while another, equally trained, equally conscientious professional may rate the same restaurant a B.

Following this line of reasoning brings us to another question: If one professional inspects the same restaurant at different times (e.g., during its busiest times and slowest times), will the grades be the same? Even though environmental health professionals are trained to discern the differences between a chronic condition and a recently created "mess," the timing of inspections still constitutes a significant variable affecting the consistency of grades. An inspection conducted immediately after the day's cleaning may yield a far better grade than one that occurs after the afternoon rush hour.

Food preparation techniques also play a role. Retail facilities that prepare "fast food" or use relatively simple techniques may have less opportunity for time-temperature violations and faster turnover of the food product. Therefore, such places may tend to receive higher grades than restaurants that use slower and more complex preparation techniques. The risk is arguably lower with faster, simpler techniques. Conceivably, we might even raise the average grade in an area by permitting only fast-food restaurants. Is this what consumers really want? Moreover, do consumers incorporate this variable into their evaluation of grades?

The size of the establishment also deserves consideration. The example of a small liquor store located next to a large market should demonstrate the point. A complex operation at a market that includes produce, meat, deli, fish, and hot-food preparation areas may yield many small violations and receive a B, while the liquor store with an operation of no complexity may receive an A even though the owner may be less committed to food safety than the owner of the larger store. This situation presents a dilemma: The large operation may indeed represent a higher risk because more things can go wrong in a complex operation; on the other hand, a single grade applied to a complex facility may not reflect that there are well run and poorly run units within the same market.

Time allotted for the inspection certainly can affect grades--more time for inspection means more time to lower the total score. Inspection time is also, however, a function of agency budgets. Policy makers may not realize that by trimming the budget they may raise the average grade for restaurants in their jurisdiction. Then again, some of them may understand all too well! In either case, a less comprehensive inspection raises the chance that individual preferences by inspectors will skew the results.

The grading system also may not account for variations in the degree of violation. For example, food kept at 130[degrees]F may pose a risk of foodborne illness and may constitute a violation, but the same food kept at 85[degrees]F would pose a greater risk. Nevertheless, some grading systems assign the same point deductions.

All of these variables serve as prelude to the next question.

Do Restaurant Inspections Measure Risk?

The critical question is not whether restaurant inspections can predict food-related illness. Of course they can! The more relevant question is how reliably can a single restaurant inspection predict food-related illness? Several additional variables influence the answer. First, some grades are based on reinspections. Such reinspections are usually by appointment, while random inspections are not. Second, postings of grades may not indicate whom consumers should contact for more detailed information. Third, the number of inspections per year may vary, which affects the relevance of the grade. The time that has passed between inspection and current conditions may vary from minutes to months. Fourth, even with frequent inspections, grades reflect a snapshot of the operation on a given day. If the sanitation of the facility drops on the day of the inspection, the restaurant is "branded" with a lower grade until reinspection. As an anonymous colleague of ours put it, one day is not every day. A more complete pic ture would be to post a history of prior grades and inspections.

Thus, grades are not always dated or archived, nor do the postings always give phone numbers for the relevant agency, nor do they indicate whether the score resulted from a random inspection or a scheduled reinspection. Even if all these pieces of information were provided, it is doubtful whether consumers would read them.

By contrast, a HACCP-based program divides restaurants--and even sections of restaurants--into high-risk, medium-risk, and low-risk areas, based on a knowledge of food safety. What we subsequently find from continuous monitoring, in terms of reported food-related illness, is therefore a more relevant measure for risk-based systems. A HACCP system typically relies on more than one inspection report. Thus, a sign indicating "this restaurant operates under HACCP" would, theoretically, be a more accurate indication of risk. Unfortunately, the public does not yet universally recognize the HACCP approach--nor, indeed, does the whole range of public health professionals.

One study recently has been cited to support the proposition that restaurant inspections predict food-related illness [20]. The study found correlations between inspection violations and reported food-related illness. The design of the study, however, was retrospective, which meant it could not measure incidence and therefore did not measure risk. This point is confirmed by the relatively low priority that the data gave to time-temperature violations [21]. Also, the study was based on single inspections rather than cumulative information.

Despite the contradicting literature, grading systems may be understood as implying that restaurant inspections measure the risk of food-related illness. This inference can be called into question for at least one fundamental reason: as discussed above, restaurant inspections are inherently inconsistent [21]. This assertion has little to do with the competence of environmental health professionals. On the contrary, it is the dynamic nature of restaurant operations that makes them so challenging to grade. Because errors in food handling can occur continuously, sometimes within short periods of time, the specific timing and strategy of an inspection can yield differences even among the most competent and conscientious professionals. We sometimes forget that standardized forms do not guarantee standardized inspections, and standardized inspections do not guarantee consistent measures of risk. Thus, the authors feel that the question of how well a grade predicts risk is open to much more intensive investigation.

Inconsistency is not, however, limited to environmental health professionals. A study at community colleges asked biology instructors to rate 15 of the most commonly known microbial agents according to the Centers for Disease Control and Prevention-National Institutes of Health (CDC-NIH) biosafety classification system. The criteria were defined on the questionnaire, and the participants were free to use any of their personal references for the study Nevertheless, the instructors were unable to correctly identify the biosafety level a majority of the time [22]. It should be emphasized that the biology instructors overwhelmingly erred on the side of safety by giving more conservative answers. Moreover, adequate explanation of the CDC-NIH biosafety classification system is not readily available in most microbiology textbooks. Nevertheless, this finding has important ramifications for the environmental health profession. If biologists cannot consistently grade the risks presented by specific microbial agents, i t is all the more difficult for the public to assess grades received by diverse retail food facilities.

Even if we ignore these issues, the grading of food facilities is clearly inconsistent among counties. Some jurisdictions give Ds, while others do not [23]. Some jurisdictions assign As to facilities that score 85 and above, and others require a score of 90 or above [24]. Some jurisdictions follow the Food and Drug Administration's model Food Code in scoring; others do not [25]. In some cases, merely crossing the county line can change the meaning of a grade. Finally, a single point may separate an A from a B establishment-and the significance of this difference from the standpoint of risk is highly questionable.

What Are the Alternatives?

The effectiveness of grading systems must be judged in relationship to the alternatives. For example, restaurant closings are often published in local newspapers. So, if the concern is protecting the public against food facilities that represent imminent hazards, closings would appear to be a superior strategy.

Another alternative is the use of the Internet to publish more detailed information about previous inspections of restaurants. If the concern is to provide easy public access to such information, the Internet would appear to be a superior approach. Accompanying educational information of high quality already exists on the Web [26]. Not everyone has easy access to the Internet, however, which brings us to the telephone. Informal though it may be, a phone call to the local jurisdiction may offer the public a relatively quick and easy way to gain information. Yet another alternative involves implementation of the Freedom of Information Act, whereby any citizen has the right to obtain detailed information on inspection reports of a given food facility [27]. Legal information about a facility--information even more complete than can be found on the Internet--can be accessed legally. A more systematic approach might apply a taxonomy of risk communication techniques [28].

If the issue were simply one of information, or access, or protection against unsafe food facilities, numerous alternatives would appear to fit the bill. All the suggestions outlined above, however, ignore the point that the call for a grading system derives from more complex motives.

Recommendations

So, are grading systems effective? The answer depends on many complex variables. By examining the diverse and sometimes conflicting purposes behind such systems, we may gain a richer insight about how to respond to public concerns. Nevertheless, the literature points to four fundamental recommendations:

1. A grade means different things to different people. Depending on culture, training, agenda, and the perceived purpose of the grading system, different members of the public can interpret inspection grades in dramatically different ways. Therefore, the authors believe there is a critical need for a more thorough evaluation of the similarities and differences in public and professional perceptions of grading systems.

2. The success of any grading system cannot be determined by a single measure, whether it be a measure of compliance, perception, epidemiological evidence, or cost-effectiveness. Assessment based on a wide spectrum of goals representative of community interests is the more accurate, though more difficult, task.

3. A triggering event such as media coverage may initiate changes. To better respond to trigger events and to diverse public concerns, environmental health professionals must view the issues in the broader perspective of policy analysis. By conducting a more thorough policy analysis, we are in better position to serve our diverse communities.

4. Over the last decade, there has been a steady stream of calls to change the nation's food safety system and to upgrade it to a more risk-based system [29-31]. Accordingly the authors recommend a national forum to discuss grading systems for retail food facilities. Such a forum could address inconsistencies among jurisdictions, recount success stories as well as disasters, and establish a network to assist local jurisdictions with issues as they arise [32]. The authors believe NEHA is well suited for such a task.

Clearly, much work remains to be done. The authors are actively engaged in examining these questions, and they look forward to presenting follow-up reports.

Corresponding Author: Owen H. Seiver, R.E.H.S., Dr.P.A., Associate Professor, Environmental and Occupational Health Program, California State University, Northridge, CA 91330.

REFERENCES

(1.) Kurtzweil, P. (1993), "Milestones in U.S. Food Labeling," FDA Consumer, 27(4):10.

(2.) Schlosser, J. (1997), "KCBS-TV Bites Back in Los Angeles: Report on Restaurant Health Violations Is Talk of Town," Broadcasting & Cable, 127(49):35.

(3.) Mehta, S.(1999), "Orange County Considers Grading Restaurants," Los Angeles Times, Orange County Edition, May 11, p. B1.

(4.) Pasco, J.O., and D. Reyes (1999), "Restaurateurs Get Help in Grade Fight: Orange County Officials Gave Food Industry Agents Tips on How They Might Get Evaluation System Scrapped," Los Angeles Times, August 13, p. B1.

(5.) Mehta, S. (1999), "Enforcing the Letter of Restaurant Law Health," Los Angeles Times, Orange County Edition, May 12, p. B1.

(6.) Sinclair, K. (1999), "Dining Out on Hygiene," South China Morning Post, March 8, p.19.

(7.) Kaur, K. (1999), "Wider Grading System?" The Straits Times (Singapore), October 22, p. 46.

(8.) Wiant, C.J. (1999), "Scores, Grades, and Communicating About Food Safety," Journal of Environmental Health, 61(9):37-38.

(9.) Bloom, A. (1998), "Food Safety--Again?" Journal of Environmental Health, 60(7):4.

(10.) Hadden, S.G. (1986), Read the Label: Reducing Risk by Providing Information, Boulder, Colo.: Westview Press.

(11.) Fielding, J.E., A. Aguirre, L.E. Frias, and M.C. Spear (1999), "Making the Grade: Changing the Incentives in Retail Food Establishment Inspection," American Journal of Preventive Medicine, 17(3):243.

(12.) National Research Council (1989), Improving Risk Communication, Washington D.C., National Academy Press.

(13.) Slovic, P. (1987), "Perception of Risk," Science, 236:280-285.

(14.) Smith, M.V. (1990), "Can Public Information Programs Affect Risk Perceptions?" Journal of Policy Analysis and Management, 9:41-57.

(15.) Moore, J. (1990), "Lists and Numbers: Effects on Public Policy?" Health and Environment Digest, 4(2):1-3.

(16.) Lave, L., and E. Males (1989), "At Risk: The Framework for Regulating Toxic Substances," Environmental Science and Technology, 23:386-391.

(17.) Graham, J.D. (1989), "Communicating About Chemical Hazards," Journal of Policy Analysis and Management, 8(2):307-313.

(18.) Rand Corporation (1981), Analysis of Alternative Design for Leaflets, Santa Monica, California: Rand Publishing.

(19.) Fischer, A. (1987), "Risk Communication: Getting Out the Message About Radon," EPA Journal, 13(6):27-31.

(20.) Irwin, K., J. Ballard, J. Grendon, and J. Kobayashi (1989), "Results of Routine Restaurant Inspections Can Predict Outbreaks of Foodborne Illness: The Seattle King County Experience," American Journal of Public Health, 79(5):586-590.

(21.) Hatfield, T.H. (1989), "Restaurant Inspections May Not Predict Outbreaks of Foodborne Illness," American Journal of Public Health, 79(12):1678.

(22.) Robinson, C.L., and T.H. Hatfield (1995), "Labeling of Microbial Risks," Journal of College Science Teaching, 24(6):407-409.

(23.) Yoon, EM. (1998), "Valleywide Stricter Restaurant Grading Proposed," Los Angeles Times, Valley Edition, September 24, p. B3.

(24.) "Five Star Lift" (1999), St. Louis Post-Dispatch, October 25, p. B2.

(25.) Emmanuel, B.P (1995), "Grading a Food Establishment (Percentage Scoring System for Food Establishment Sanitation)," Journal of Environmental Health, 58(3):20.

(26.) "FDA.GOV" (1999), FDA Consumer, 33(5):32.

(27.) Freedom of Information Act (1966), 5 U.S. Code [ss]552.

(28.) Hatfield, T.H. (1994), "A Risk Communication Taxonomy for Environmental Health," Journal of Environmental Health, 56(8):23-28.

(29.) Robinson, R.A. (1997), Food Safety: Fundamental Changes Needed to Improve the Nation's Food Safety System, GAO "Reports, GAO/TRCED-98-24, Washington, D.C.: General Accounting Office.

(30.) Food Safety: A Unified, Risk-Based Food Safety System Needed (1994), GAO Reports, GAO/T-RCED-94-223, Washington, D.C.: General Accounting Office.

(31.) Hatfield, T.H. (1991), "The Failure of Sanitarians," Journal of Environmental Health, 53(5):23-25.

(32.) Powell, DA., D.M. Alves, M.W Griffiths, A. Lammerding, and J. Lynch (1999), "Evaluation of Electronic Information Sources to Identify Food Safety issues for Risk Management and Communication: The Creation and Assessment of the Food Safety Network (FSnet)," Dairy Food, and Environmental Sanitation, 19(9):618.