A systems-based food safety evaluation: an experimental approach.

Introduction

Traditionally, government food safety programs have used, as the basis for their work, an inspectional or code-based approach. This approach involves comparing a list of "dos and don'ts" to what is actually observed in any particular food service establishment. Many of these lists are derived from the various versions of the food safety model ordinances produced by the U.S. Public Health Service, starting with the 1935 version, "An Ordinance Regulating Food and Drink Establishments." Once inspectional observations are recorded, they become alleged violations of the applicable regulation. Educational (informative) and enforcement (coercive) methods are used in an attempt to eliminate or mitigate the list of violations.

[ILLUSTRATION OMITTED]

Anecdotal evidence of dissatisfaction with this traditional approach is to be found in the hundreds of presentations, seminars, courses, and meetings that over the last 20 or more years have discussed the drawbacks of inspection and possible alternatives. A series of articles by Dr. Frank Bryan, formerly of the Centers for Disease Control (CDC), discussed the need for an inspection process that better identifies and focuses on the factors that cause foodborne outbreaks (Bryan, 1978). Dr. Bryan also wrote about a new concept, the hazard analysis and critical control point (HACCP) approach, which was developed for the space program by the Pillsbury Company and the U.S. Military's Natick Laboratories (Bryan, 1981).

In recent years, many agencies and jurisdictions, including the Food and Drug Administration (FDA) and the Food Safety and Inspection Service (FSIS) of the U.S. Department of Agriculture (USDA), have begun trying to apply HACCP principles to food safety inspections. The HACCP concept involves charting out the flow of foods in any operation, identifying the hazards associated with those flows, and looking for controls for certain critical steps. This approach, however, designed as it originally was for large processors, has proven difficult to use and sustain, both by small retail operations and by regulators.

The National Park Service (NPS) Public Health Program (PHP) is charged with acting as a consultant to the nation's parks system in matters of public health. PHP has used the same traditional approaches as other food safety programs, and at the core of this effort is the FDA model food code (FDA, 2001). Over the last two years, PHP has been exploring a concept that attempts to combine the conceptual aspects both of the HACCP methodology and of an older idea, General Systems Theory, first proposed in the 1940s by biologist Ludwig von Bertalanffy (Bertalanffy, 1968).

Bertalanffy's work stated that "the whole is more than the sum of its parts." Systems theory emphasizes the need to understand the underlying interactions of all of the forces that make up any system. He further stated that all systems have what are called natural "set points," essentially outcomes, predetermined by the nature of these underlying forces. Making changes within any complex system can prove difficult to impossible unless a great deal is known about these underlying forces. Furthermore--and perhaps just as important to food programs--unless the deeper systems factors are dealt with or changed, any system tends to "reset" to its original set point. Food inspectors who have cited violations often see them corrected only to find a few months later that the same problem is back again. While there is no known specific research on this issue, it is possible that this anecdotal "violation return" could result from a lack of dealing with underlying causes, and that the system gradually resets.

This paper proposes that food service operations can be viewed as systems, with inputs, process, outputs, and feedback. As systems, they should have the general characteristics of any system, including complex underlying factors that drive how things work and why, and natural set points (outcomes) determined by these systems forces (Figure 1). In a food service operation, the underlying factors that affect the systems outcome (in this case, the relative risk to consumers from the foods produced) might be placed into five organizing categories:

* foods and their inherent properties,

* people,

* equipment,

* process, and

* economic issues.

In 2001, PHP set out to develop this combination of concepts into a realistic field approach to the assessment of food service establishments and to use information gathered by these methods in efforts to improve food safety. Over the course of the last two years, an experimental systems-based methodology has been drafted, developed, and tested by NPS/PHP. This experimental methodology is evolutionary rather than revolutionary. In designing its approach, NPS has built on the Food Code and on past inspection techniques, and while using the HACCP principles to help determine the degree of control over key food safety flows, this approach goes beyond the HACCP concept, "drilling" below the flow charts and into the "systems forces" described by General Systems Theory.

This food safety evaluation methodology has been designed for and is being piloted in a variety of retail food service operations, including full-menu restaurants, quick-service restaurants, grocery stores/deli operations, and specialized food service operations found largely within the National Parks system.

During the course of this pilot effort, program personnel, operators, and NPS specialists who oversee the contracts of these food service operations have all been pleased by its use. Anecdotal information indicates that the approach is a natural one for both evaluator and operator and helps them to focus on important food safety management issues. It is hoped that, over time, an understanding of the deeper reasons behind the set points of kitchen outcomes will lead to effective, real, and sustainable improvements in food safety. Informal, early success of this pilot program has led NPS to consider expanding the use of these ideas and to consider committing resources to the formal evaluation of the effectiveness of this methodology. Because of the early, positive feedback, NPS wishes to share these ideas with other programs in the hope that others will consider the exploration and evaluation of this direction.

[FIGURE 1 OMITTED]

Experimental, Systems--Based Food Safety Evaluation--Draft Methodology

Step 1: System Description

The first step in this method is to gather background information about the system to be evaluated. All other information about the system needs to be placed in the overall context in which the operation takes place. Because there are outside forces (systems operate within larger systems) that affect an operation, these "macro-forces" need to be described.

During any initial visit or when any operation changes management, asking a series of questions will help to build a description of the establishment. These questions were designed to assist understanding of the overall input and feedback forces that might influence how a kitchen operates and why it operates the way it does. A simple summary of this information is documented at the top of the first page of a food safety evaluation report. (Note: many jurisdictions already using HACCP-based approaches will find this first step very similar to current practice.) The series of questions to be asked is as follows:

* Does the operation have an official, written set of food safety policies? Does the operation have a formal policy on employees working while ill? If not, how is this issue handled?

* Describe the general nature of your operation, your type of service, hours of preparation and service, any off-site service, number of employees, etc.

* What is the educational and experience background of management and staff?

* Is there an initial or new-employee training program? Describe.

* Does the operation conduct any internal inspection/QA activities? If so, describe.

* Describe the nature of any external audit/inspection activities.

* Describe any economic pressures, rewards, reports, etc.

* Is there any formal form for feedback from customers (comment cards, etc.)?

* Is there any feedback that is received from the park, and if so what is the nature of the feedback?

* Describe the management structure and operation. What type of supervision is given to front-line employees?

* Do seasons or other factors cause any variations in the operation?

* Describe the pressures that you as a site manager feel. What types of factors help you or cause you to make decisions?

Step 2: Process Evaluation

To determine the hazards associated with the operation and to establish the degree of control over these hazards, a process evaluation is conducted.

Food service menu items can be divided into three primary flows or processes, with each process entailing different preparation techniques, hazards, and food safety controls. The three processes are listed in the FDA draft document, "Managing Food Safety: A HACCP Principles Guide for Operators of Food Establishments at the Retail Level" (FDA, 1998). According to this document, the first process category comprises foods that are not cooked, such as salads that never receive any cooking or that do not receive any further cooking in the restaurant. The second process category is cook and serve. This category includes foods such as hamburgers that are cooked and served directly to consumers with only one pass through the food safety temperature danger zone (41[degrees]F to 135[degrees]F). The final category is called complex flows and involves more than one pass through the danger zone. Soups and stews usually belong to this category, and any other foods that require extensive preparation and handling involving many steps over longer time periods.

Most of the food items in the example menu in Table 1 fit into the category of cook and serve. Some food items include elements that extend preparation to the day before service; this circumstance would place them in the complex-flow category. A few items on the menu are foods that are not cooked.

Not only can foods on any menu be separated into these three process categories, but these categories also reflect natural lines or divisions for classifying different types of hazards, outbreaks, and controls (Table 2). While not all outbreaks are covered by this method, the vast majority are. One advantage that this approach has over classic HACCP protocol is that it provides for a science-based shortcut to the tedious and difficult task of producing a flow chart and hazard analysis for each and every menu item (a medium-sized restaurant menu may contain 150 menu items).

After the menu is divided into these categories, data must be collected about the degree of control that an operation has over each process. When the menu items in each process, the hazards inherent to each process, and the necessary controls are all known, observations can be made in the kitchen to assess the overall management of the three categories. Using a combination of observation, measurement, and interviews, the evaluator makes notes about actions, behaviors, and conditions that result in control and those that pull the process away from or out of control. For example, an employee places a food into an ice bath to cool, but the next shift comes in shortly after that action and takes the product to a walk-in without checking the temperature. This action lessens the control that this operation has over the rapid chilling of foods being prepared in a complex flow. A positive example might be found in an interview with a line employee who states that the supervisor makes rounds twice a shift to observe handwashing and discuss handling with employees. This is a positive force, adding to the degree of control that the establishment has over foods that are not cooked.

Step 3: Exploration of Underlying Cause and Effect

Beyond the HACCP principles, the General Systems Theory states that understanding the underlying reasons or system forces that cause things to operate the way they do is vital to any effort to change a system outcome or set point. After determining the degree of control there is for the three basic flows discussed above, it is necessary to "turn them sideways and drill down" into the deeper "whys" of the system. On the basis of the information obtained in Step 2, individuals seen in control of or not in control of key issues are interviewed in an attempt to understand why they chose the actions that were observed. Cause-and-effect diagramming is the mental model for these informal interviews. Starting with the behavior and working backward, interviewers ask employees questions that include some variation of "why." For example, if a cook has been observed washing his or her hands at appropriate times, the question sequence might look like this:

* Evaluator: I noticed that you were washing your hands after handling the raw chicken. Can you tell me why?

* Employee: Because if I didn't, the supervisor would say something.

* Evaluator: Why is that?

* Employee: Because they have a policy of washing hands between raw meats and other foods. Supervisors are required to check on this.

* Evaluator: Interesting! When you can't wash your hands, what keeps you from being able to?

* Employee: Well, I usually do, but ... I suppose when it gets really busy,... that becomes almost impossible.

* Evaluator: Any examples?

* Employee: Sure, the cooks line at dinner, and I suppose when I'm working in the pantry and we get slammed, I ... well, just have to go, go, go.

This approach to interviewing takes some time to learn, and a combination of knowledge about how to ask questions and experience in what gets useful information is important. Asking questions, like any other activity, is a skill that requires practice. As with the practice of other skills, paying attention to the result and using that information to readjust the questions helps in obtaining more useful information. Because gathering this deeper information is vital to the success of any systems-based approach, the authors include at this point some specific ideas.

Useful Starting Assumptions

1. People are unsure about themselves and naturally protect their standing/image with others.

2. People being asked questions assume they are being tested or evaluated.

3. The more comfortable or safe we feel the more open we are likely to be in answering questions. This sense of comfort is dependent, among other things, on whether we know the person asking the questions, what we think the information will be used for, and how we perceive that it will reflect on us.

4. Everyone views the same situation or procedure differently, so all of the above plus this fact mean that answers, even about the same procedure, will differ from person to person.

5. In posing our questions, we are uninterested in blame, but are instead very curious about the system, how it operates, and why, so that we may explore ways to strengthen it.

Types of Questions

There are many types of questions, all useful for different purposes. Given the purpose at hand, here are a few types and some explanation and examples of their use.

1. Closed Questions -- Used to get yes or no answers. These questions may start with words such as do, is, can, could, will, would, should, and shall.

a. Example: Do you cook the chicken to 165[degrees]F?

b. Use: When looking for very specific information, confirmation, commitment, or agreement.

2. Open Questions -- Used to allow for a wider range of answers and gather information. These questions commonly start with words such as how, why, when, where, what, who, and which.

a. Example: What temperature do you cook chicken to?

b. Use: To get more than a yes or no, to allow for a more expansive answer.

3. Probing Questions -- An open question that uses information already known or gathered in order to get additional information (see also Questions of Clarification).

a. Example: What temperature do you cook your chicken to?

A: 165[degrees]F.

Probing Question: Why do you cook your chicken to 165 degrees?

b. Use: To drill down below the surface answer and gain additional information or insight.

4. Direct Questions -- These questions can be open or closed but in this context will most often be open questions that include an instruction (it also helps to use the person's name).

a. Example: Tell me, Ed, how do you prepare your chicken? Just start at the beginning and step me through it.

b. Use: Allows the subject to tell you, expansively, in his or her own way about a specific subject that you are interested in. A question of this type gives the subject some instruction about how you want the story told (in this example, from beginning to end, step by step). Using subjects' names gets their attention, makes the conversation more personal, and indicates your interest in what they, personally, have to say.

5. Hypothetical Questions -- These are questions that place subjects in a situation and ask them to answer based on that situation.

a. Example: Ed, when you are not able to wash your hands, what keeps you from being able to do that?

b. Use: To invite the negative. Most people, when asked something that may incriminate or embarrass them, will give you the answer they think you want to hear or that will make them look good. By phrasing a question this way, we have "given them permission" for the negative. Questions of this type often are used when the goal is to get someone to admit to a "wrong" answer, and in this context are likely to be used to get at the why of some difficult issues.

[FIGURE 2 OMITTED]

6. Reflective Questions -- Questions that invite the subject to dig deeper, provide opinion, or expand on an answer.

a. Example: The person you are talking to is somewhat excited about a subject and is saying, "... cooling is our most difficult challenge, and I hate it!" You follow up with, "Ahh, a lot of people do, that's usually a hard one, tell me more about your struggles."

b. Use: You have obviously been listening and are following up; this kind of question shows interest in what the person has to say and builds rapport. Getting to this point can often open up vast amounts of deep information.

7. Questions of Clarification -- Can be of any of the previous types and are used to expand, clarify, or make a previous answer more meaningful.

a. Examples:

What do you mean?

How does _____ relate to _____?

Can you put that another way?

What do you think that means?

Could you give me an example?

Why do you say that?

I'm not sure I understand, can you explain that further?

What leads you to decide that?

Why do you believe that?

Why would you choose to do that?

What was your reason for _____?

b. Use: To add meaning, get below the surface, and follow up on a lot of the "why" issues.

Hints

* Don't answer the question yourself.

* Be sure to give the subject time to think and answer.

* Hold off on writing right away if you can to better relax the person.

* Avoid leading comments or body language unless used purposefully.

* Avoid asking multiple questions--you may not be aware you're doing this.

* Listen, listen between the lines, listen to body language, and listen.

Step 4: Documentation and Strengthening the System

After collecting information about the nature of the system and the factors that influence food safety outcomes, it is necessary to document this effort and to begin to design public health interventions that will strengthen the system.

At the beginning of the NPS Food Safety Report form (Figure 2) is an area in which basic background information about the establishment is documented. This summary should be simple but designed to let someone who was not there know in what operational context all of the information to follow occurs.

Any unknowns, information that was not gathered, controls that were not observed, or notes for items that need to be investigated further should be placed in the second box, titled, "Areas not evaluated this visit OR needed follow-up." Information documented on the evaluation form is designed to indicate the degree of control over key food safety issues. If the evaluator is unable to assess that control, then this section is used to note this fact (for example, few foods may be cooked during the evaluation and so cooking temperatures may not have been taken). Another way to use this section is for reminders, such as, "Interviews indicate that the evening crew may conduct cooling in ways that differ from earlier shifts--this variation needs to be further investigated during the next visit."

Box 3, "System Strengths," is used to summarize systemwide protective practices that cross process lines. For example, a well-trained manager who has a positive attitude toward food safety issues could be mentioned here. Another example of data that would be recorded in this box would be a well-carried-out training program for all employees.

Positive observations are recorded in the boxes on the lower right of the first page of the Food Safety Report. These observations are organized by the three process categories. They should be actual knowledge, documented either by firsthand interviews or through observation/measurement(s) of conditions or practices that are protective of a safe outcome.

Negative observations are recorded on the second (back) page of the Food Safety Report. These observations may take two forms, noting either violations of the Food Code or system weaknesses that may not pertain to code requirements or may be judged not to rise to the status of code violations. Text referring to actual code violations is accompanied by filling in of the boxes to the left: whether or not the violation is critical, whether the violation is a repeat from the previous evaluation, and the numerical model code citation. If the notation under "Description/Remarks" is considered to refer to a system weakness but not a code violation, then the three boxes to the left remain empty. Information about any foods discarded or actions taken in response to one of these weaknesses should be noted in this section.

Using the information about strengths and weaknesses and adding insight from the exploration of "why," discussion with the operator should result in a set of agreements about ways to strengthen the system. These should be captured in the last section, titled "Agreements/Actions to Strengthen Food Safety." They should not be simple corrections to violations such as "reheated the soup to 165[degrees]F," but should be deeper, more permanent improvements that address the reason an issue was not controlled. To obtain greater buy-in for these statements, it might be appropriate to have the operator actually word and write these statements.

Because of space requirements and because the two pages of the report are intended only as a summary of the evaluation, NPS uses an addendum page to document most of the actual data gathered during the visit. Food temperatures, food flows, handling observations, and other measurements can all be placed on the addendum page and then summarized in the report. The addendum is attached to the report and provided to the operator at the exit discussion.

The Future

Any translation of a conceptual approach such as systems-based ideas to application in the field requires compromises to ensure that the field methods are doable and practical. Attempting to understand a complex system, particularly why it operates the way it does and what the most influential underlying systems forces might be is a difficult task. The difficulties are exacerbated when one is trying to design shortcuts that will allow a realistic timeframe for an on-site evaluation. This process control design for a systems-based evaluation shows promise for improving the ability to identify food safety hazards, for more deeply understanding the underlying reasons that food safety is compromised, and for using this information to design more intelligent interventions. NPS hopes to continue to work with these ideas, expanding the initial pilot efforts and taking the next step of devising ways to measure whether this approach helps regulators and operators gain additional control over food safety issues.

TABLE 1 Sample Menu with Food Items Listed Under Different Process
Categories

Food Item                     Not Cooked  Cook and Serve  Complex Flow

Round of Rocky Mountain                                     X
  buffalo
Cornmeal-crusted river trout                                X
Corn on the cob                             X
Beans                                       X
Cornbread                                   X
Potato salad                                                X
Home fries                                  X
Coleslaw                        X
Baked rolls                                 X
Crudites platter                X
Pumpkin streusel spice cake                 X
Apple pie                                   X
Rhubarb cobbler                             X
Beverages                       X

TABLE 2 Hazards and Controls of Not-Cooked, Cook-and-Serve, and Complex-
Flow Foods

Food            Hazards               Controls

Not cooked      Cross-contamination   Exclude ill employees
                from meats and        Handwashing
                employees             No bare-hand contact with
                                      ready-to-eat foods
                                      Prevent cross-contamination from
                                      meats to other foods
Cook and serve  Possible survival     Ensure that foods are cooked to
                of disease organisms  proper temperatures
                of animal origin
Complex flow    Possible growth       Cool foods quickly
                of spore formers      Keep hot foods hot and cold foods
                or contamination      cold
                after cooking         Properly reheat
                                      Protect cooked foods from
                                      contamination

Acknowledgements: Intellectual acknowledgement is due to Carol Selman, John Sarisky, Pat Bohan, and Dr. Joe Hollowell who, along with the corresponding author, originally conceived of these ideas while all five were working at the National Center for Environmental Health at the Centers for Disease Control and Prevention.

REFERENCES

Bertalanffy, L. (1968). The meaning of general systems theory. In General systems theory: Foundations, development, applications (pp. 30-53). New York: George Braziller.

Bryan, F. (1978). Factors that contribute to outbreaks of foodborne disease. Journal of Food Protection, 41, 816-827.

Bryan, F. (1981). Hazard analysis critical control point approach: Epidemiologic rationale and application to food service operations. Journal of Environmental Health, 44(1), 7-14.

Food and Drug Administration. (1998). Managing food safety: A HACCP principles guide for operators of food establishments at the retail level. http://www.cfsan.fda.gov/~dms/hret-toc.html (Jan. 2003).

Food and Drug Administration. (2001). Food code. Springfield, VA: The National Technical Information Service.

CAPT Charles L. Higgins, R.E.H.S., M.S.E.H.

CAPT Barry S. Hartfield, R.S., M.P.H.

Corresponding Author: CAPT Charles L. Higgins, Director, Public Health Program, National Park Service, 1201 Eye St., NW, 11th Floor, Room 54, Washington, DC 20005. E-mail: [e-mail].