Assessing the effectiveness of food worker training in Florida: opportunities and challenges.

Introduction

From a public health standpoint, it is important to assess the effectiveness of food worker training programs in preventing foodborne outbreaks (McNamara, 1999). Food worker responsibilities include handling and preparing food that will be served to consumers, and most professionals agree that food worker training and education are essential in preventing foodborne disease by promoting good food safety and hygiene (Cotterchio, Gunn, Cofill, Tormey, & Barry, 1998; Riben et al., 1994).

Until recently in Florida, only food managers were mandated to be certified in food safety and sanitation. The assumption was that they would in turn train food workers employed in their establishments. In the year 2000, changes were made to Florida law requiring training of all food workers in food establishments inspected by the Florida Department of Business and Professional Regulation (DBPR). The change in training requirements, coupled with existing data on foodborne-disease outbreaks and cases compiled by the Florida Department of Health (FDOH), provided a unique opportunity to explore methods of assessing the effectiveness of food worker training requirements.

This paper discusses the challenges of measuring the effectiveness of food worker training, various methodologies that might be considered, and the results of utilizing trends in foodborne-outbreak contributing factors to evaluate Florida's recent change in food worker training requirements.

Background: Prior Efforts to Measure Food Worker Training Programs

Researchers have made numerous efforts to evaluate the effectiveness of food safety education (Mathias, Sizto, Hazlewood, & Cocksedge, 1995; Raval-Nelson & Smith, 1999; Reed, 1982). Most have focused on the effect of food worker training on critical inspection violations. A Canadian study concluded that although food-handling practices did improve and inspection scores were higher soon after training, no differences in scores were found during inspections performed six months after training (Mathias et al., 1995). Another study of the Food Safety Certification Program in Philadelphia indicated a positive impact on learned behaviors; however, the survey used to assess the effectiveness of certification training was too limited to determine the degree of application of the learned behaviors (Raval-Nelson et al., 1999).

In addition, several state and local health jurisdictions are looking at measuring the effectiveness of food worker training. The state of New York has collected data on factors contributing to foodborne illnesses for at least 10 years, but has not written any report or summary at this time. The Tacoma-Pierce County Health Department in Washington State has written a draft report on the subject using specially designed questionnaires to assess the extent to which knowledge about key food safety concepts was retained subsequent to training. The state of Minnesota is also conducting a study on food worker training.

Other colleagues responsible for investigating and preventing foodborne-disease outbreaks indicate an ongoing interest in assessing the effectiveness of food worker training. A presentation given at NEHA's Annual Educational Conference in Atlanta on July 1, 2001, was titled, "What Should Be Measured to Determine if a Food Safety Education Program Is Effective?" Furthermore, the Food and Drug Administration (FDA) has attempted to establish national baseline figures on the occurrence of foodborne-disease risk factors in its Report of the FDA Retail Food Program Database of Food borne Illness Risk Factors (FDA, 2000).

How best to evaluate the effectiveness of food worker training programs has not been established. Some studies have looked at utilizing the findings of restaurant inspections to predict foodborne-outbreak occurrence. These studies have reported mixed results. While the Washington study in Seattle-King County did find an association between outbreak occurrence and critical violations (among other factors) (Irwin, Grendon, & Kobayashi, 1989), a later study in Miami-Dade County, Florida, showed that inspections in that county did not reliably predict the occurrence of outbreaks (Cruz, Katz, & Suarez, 2001). The variety of prior approaches and varied results attest to the difficulty in measuring the effectiveness of food worker training.

Methods

Seven methods of assessing food worker training effectiveness and the limitations of each method were considered by the authors. Only the last method listed, Method 7, was utilized in the authors' attempt to assess the benefits of Florida's recent change in training requirements; however, the other methods may be appropriate and useful in other situations. All seven are listed here for review:

1. Comparison of the number of foodborne-illness outbreaks before and after implementation of food worker training. Limitation: Because outbreaks have a variety of causes and there is variability in outbreak reporting and detection, this approach lacks specificity.

2. Comparison of the number of cited critical violations in restaurants before and after implementation of food worker training. Limitation: Baseline data on violations before implementation of training were not available to the authors.

3. Correlation of the number of foodborne-illness outbreaks with the number of food workers trained after implementation of training. Limitations: food worker mobility and lack of consistent records on training of food workers statewide.

4. Comparison of the occurrence of outbreak contributing factors in food establishments inspected by DBPR (restaurants), where training is required, and the occurrence of these factors in other food establishments (e.g., schools, institutions and grocery store delicatessens), where the new 2000 food worker training is not required. Limitations: different workforce makeup in the various types of food establishments (restaurants generally having a less stable workforce than other food service establishments), differences in education and language capabilities of the workforce in various establishments, and differences in types of foods and food preparations required in different types of food establishments.

5. Comparison of trends in foodborne-illness outbreaks and cases associated with specific pathogens (e.g., hepatitis A, E. coli O157: H7, S. Typhi, Shigella spp., Salmonella spp., Norovirus, Staphylococcus) occurring before and after implementation of new training requirements. Limitations: Again, this measure is not specific enough to assess the impact of food worker education and involves many confounding factors.

6. Comparison of the training compliance rate of food establishments having outbreaks with the training compliance rate of food establishments not having outbreaks. Limitations: Current data regarding the identity of a restaurant involved in an outbreak are not available electronically, and manual abstraction is required.

7. Comparison of the trends, before and after implementation of training, in contributing factors associated with foodborne outbreaks and cases. The authors chose this method of analysis to assess the new 2000 worker training requirements in Florida and believe it offers the best comparison available at this time. The reasons this approach was chosen and its limitations will be covered in the discussion section of this paper.

Changes in Food Worker Training Requirements in Florida

Since 1988, it has been required that food service managers in DBPR-licensed food service establishments be trained in food safety standards (Florida Statutes, [section] 509.039). On January 1, 1998, further legislation was promulgated requiring the training of all public food service employees in professional hygiene and foodborne-disease prevention (Florida Statutes, Section 509.049). These training requirements delineated the areas of food safety and hygiene that must be covered in training. Food service establishments regulated by the Florida Department of Health (FDOH) and the Florida Department of Agriculture and Consumer Services are not covered by this requirement. DBPR establishments include over 40,000 Florida restaurants. While it was required that all food service workers be trained, it was assumed that the training would be provided by certified food managers working in the same establishment. No documentation or tracking of food worker training was required at that time.

In 2000, the Florida Legislature enacted legislation that further defines training requirements for all food workers in DBPR-licensed food service establishments (Florida Statutes, [section] 509.049(6)c). In the state of Florida, DBPR establishments include mostly restaurants and caterers, but not grocery stores, food processors, or institutional food service settings. These changes in the Florida Statutes require a food safety training certificate program for food service employees to be administered by a private, nonprofit provider chosen by DBPR's Division of Hotels and Restaurants. In addition, any food safety training programs established and administered to DBPR food workers before the effective date of the new requirements (July 1, 2000) had to be reviewed and approved by the Division of Hotels and Restaurants. By January 1, 2001, all currently employed DBPR food service workers had to receive food worker training by an approved certification program. Specifics of training requirements can be obtained from DBPR's Web site (Florida Department of Business and Professional Regulation, 2000).

In general, an approved certification program for food workers requires that employees be trained in 1) food safety criteria stated in Florida laws and rules, 2) major risk factors considered in foodborne outbreaks, and 3) the FDA Food Code's five intervention strategies. The Food Code's five intervention strategies focus on the following areas:

* demonstration of knowledge,

* employee health controls,

* controlling hands as a vehicle of contamination,

* time and temperature parameters for controlling pathogens, and

* consumer advisory.

Employees hired after November 1, 2000, had to receive certified training within 60 days of being hired. The revised law stated that certification is good for three years. After three years, an employee in a DBPR-licensed food service establishment must be retrained and recertified.

Available Data in Florida Regarding Food-borne-Outbreak Contributing Factors

FDOH has had a unique Food and Waterborne Disease Program in place since 1994, offering surveillance, investigation, and training assistance to the state's 67 county health departments. Data are collected from each of the 67 counties on a monthly basis by nine regional environmental epidemiologists employed by the DOH Bureau of Community Environmental Health. Foodborne-outbreak data are compiled on a quarterly basis to calculate Quarterly Performance Measures and yearly to produce an Annual Report on Food and Waterborne Diseases.

Since 1997, the Food and Waterborne Disease Program has been systematically collecting data on factors that contribute to foodborne-disease outbreaks. Contributing factors associated with foodborne outbreaks and cases that were consistently investigated and documented from 1997 through 2003 were used by the authors to evaluate the effectiveness of the new food worker training implemented in 2001.

The number of foodborne-disease outbreaks and cases associated with each of 17 contributing factors were obtained from annual reports produced by the Florida Department of Health. The 17 specific contributing factors chosen for comparison in this report are derived from the Center for Disease Control and Prevention's (CDC's) identification of five broad risk factors related to food employee behaviors and food preparation practices in retail food service establishments, and from the risk factors currently in use by the CDC Electronic Foodborne Outbreak Reporting System (CDC, 2003; Olsen, MacKinon, Goulding, Bean, & Slutsker, 2000). CDC's five broad risk factors are as follows:

* improper holding temperatures;

* inadequate cooking, such as undercooking of raw shell eggs;

* contaminated equipment;

* food from unsafe sources; and

* poor personal hygiene.

Analysis

The numbers of foodborne outbreaks and cases associated with each of the 17 contributing factors were summarized into two categories: the four years (1997-2000) before implementation of food worker training and the three years (2001-2003) after implementation of training. (In the context of this discussion, the term "associated contributing factor" means that the contributing factor was documented to be present in the facility at the time of the outbreak.) A total of 1,001 outbreaks of foodborne disease were found to be associated with one or more of the 17 contributing factors in the four years before training implementation; 581 outbreaks were found to be associated with one or more of the contributing factors in the three years after training implementation. These totals were used to determine the rate of outbreaks associated with each contributing factor per 1,000 outbreaks. A total of 5,651 cases of foodborne disease were found to be associated with one or more of the contributing factors before training implementation, and 3,582 cases were found to be associated after training implementation. These totals were used to determine the rate of cases associated with each contributing factor per 1,000 cases.

Using the total number of foodborne-disease outbreaks or cases associated with each contributing factor per 1,000 outbreaks or cases, the authors analyzed the rates using a Z-test for the equality of two proportions (binomial distribution). This test for difference of proportions was applied to each contributing factor to determine whether a significant change in the rate of outbreaks or cases associated with that contributing factor occurred after training was implemented. The results of the analysis are summarized in Table 1 and Table 2.

Results

Table 1 summarizes the number and rate of foodborne outbreaks, by contributing factors, before and after implementation of training in DBPR-inspected facilities. The p-values indicate whether there was a significant change in the rate of outbreaks associated with each contributing factor after training began. A statistically significant increase occurred in the proportion of outbreaks due to eight (47 percent) of the 17 contributing factors. These eight were as follows: liquid/semi-solid mix of potentially hazardous food, raw or lightly cooked food, raw product contaminated by animal or environment, bare-hand contact, slow cooking, inadequate cold-holding temperature, infected food handler, and storage in a contaminated environment. A significant decrease occurred in the proportion of outbreaks due to three (18 percent) of the 17 contributing factors. These three were as follows: insufficient time or temperature during cooking, insufficient time or temperature during hot-holding, and polluted source. In six categories (35 percent) there did not appear to be any statistically significant change.

The total number of foodborne-disease outbreaks in restaurants associated with one or more of the contributing factors that we studied decreased from 1,001 (in 1997-2000) to 581 (in 2001-2003). Before training, the average annual occurrence of foodborne-disease outbreaks associated with one or more of the contributing factors before training was 250 per year (in 1997-2000); after training it was 194 per year (in 2001-2003).

Table 2 summarizes the number and rate of individual cases, by contributing factor, before and after implementation of training in DBPR-inspected facilities. The p-values indicate whether there was a significant change in the rate of cases associated with each contributing factor after training began. In 2001-2003, a statistically significant increase was noted with respect to 10 (59 percent) of the 17 contributing-factor categories. A decrease was noticed in the proportion of cases caused by six (35 percent) of the contributing factors, and no change occurred in cases associated with one category (raw product contaminated by animal or environment).

The total number of foodborne-disease cases in restaurants associated with one or more of the contributing factors that the authors studied decreased from 5,651 cases (in 1997-2000) to 3,582 cases (in 2001-2003). Taking an average, the annual occurrence of foodborne-disease cases associated with one or more of the contributing factors before training was 1,413 per year (in 1997-2000); after training it was 1,194 per year (in 2001-2003).

Discussion

It is generally accepted that proper education is a key to overcoming many adverse health outcomes, and this premise also applies to improving food safety for consumers. At the USDA-sponsored conference on food safety education in September 2002, attendees from 48 states and 10 countries were surveyed in several areas. Food worker training was considered the number-one priority with respect to food safety by the over 500 attendees surveyed.

Measuring the public health impact of food worker training is challenging, and many previous investigators have met with limited success. In this paper, the authors have explored the challenges of measuring the effectiveness of food worker training, as well as the various methodologies that might be used to assess the effectiveness of food worker training. After considering the various possible approaches, the authors chose to compare the occurrence of food worker contributing factors associated with foodborne outbreaks and cases of foodborne illness before implementation of training requirements on January 1, 2001, with the occurrence of contributing factors involved with outbreaks and cases occurring after January 1, 2001, in food service establishments licensed by the Florida DBPR.

Reasons for selecting contributing factors as a measure of the public health impact of food worker training include the following: 1) availability of these data in Florida, 2) the focus that food worker training places on these contributing factors, 3) the acceptance of the importance of these factors in foodborne outbreaks, and 4) lack of previous studies on this approach to evaluating training effectiveness.

The data in Table 1 and Table 2 summarize the findings of epidemiologic investigations subsequent to foodborne-disease outbreaks and cases and clearly indicate that food worker contributing factors are often documented to be involved in the occurrence of foodborne illness. The data from the authors' preliminary study and analysis illustrate that the relative incidence of many documented contributing factors has actually increased, while the incidence of others has decreased subsequent to the initiation of food worker training.

The overall annual occurrence of outbreaks and cases associated with one or more of the contributing factors the authors studied decreased after the implementation of the new training requirements in Florida. These data, however, have to be interpreted with caution, as outbreaks have a variety of causes and there is variability in reporting and detection of outbreaks and cases. Subsequent to the implementation of training, there were actually more outbreaks and cases involving multiple contributing factors. Again, there could be a variety of reasons for this outcome. For example, epidemiologists may have been more attuned or better trained to identify and document contributing factors in the later years of the study. In addition, data utilized in this analysis were derived from statewide data, and specific data were not available to indicate which food establishments were in compliance with training.

On the basis of the data available and the results of the authors' analysis, it is not possible at this time to make definite conclusions about the overall effectiveness of the newly implemented food worker training program. The data give information about the relative frequency of occurrence of contributing factors associated with outbreaks or cases. It does become apparent that following the trends of contributing factors associated with outbreaks may be a useful evaluation tool. Finding a negative trend in the incidence of contributing factors may be as useful as finding a positive trend, because both serve as guides for altering training material and methods and focusing training on needed areas. Utilizing contributing-factor data in conjunction with overall numbers of outbreaks and cases may provide additional useful information in the evaluation of training effectiveness.

More time is needed to assess whether this current approach is a useful method for tracking overall training results, or whether one of the other approaches described in this paper will be a better tool.

Regardless of the method chosen, there will be challenges in determining the success or failure of training. For example, restaurants have a special problem with maintaining an educated workforce because they have an extremely high turnover rate in comparison with other industries. While they are training all their staff within 60 days of hire, some establishments may also have to replace 75 percent of their staff within a few months' time. Training must start again with each new employee. This situation adds to the challenge of verifying the compliance rate of training programs in food establishments statewide.

Conclusion

In summary, common sense suggests that food worker training programs will improve food handler practices and decrease the risk of foodborne illness and foodborne outbreaks. Assessing the public health impact of food worker training will continue to be an important public health issue. A variety of challenges and limitations are involved in measurement of the overall effectiveness of food worker training programs. Following trends in the contributing factors associated with foodborne-diseases outbreaks and cases may be a useful approach for determining the effectiveness of food worker training, and it may be helpful in directing ongoing training efforts. Knowing the trends in contributing factors can help to determine areas of food worker safety that need emphasis in training programs. Further work needs to be done to establish the methods and approaches that are most useful for assessing the effectiveness and hence the public health impact of food worker training.

Corresponding Author: Roberta M. Hammond, Food and Waterborne Disease Coordinator, Florida Department of Health, 4052 Bald Cypress Way, Bin #A08, Bureau of Community Environmental Health, Tallahassee, FL 32399-1712. E-mail: [e-mail].

REFERENCES

Centers for Disease Control and Prevention, Foodborne and Diarrheal Disease Branch. (2003). Investigation of a foodborne outbreak reporting form. http://www.cdc.gov/foodborneoutbreaks/report_f/reporting_form.doc (4 Oct. 2004).

Cotterchio, M., Gunn, J., Coffill, T., Tormey, P., & Barry, M.A. (1998). Effect of a manager training program on sanitary conditions in restaurants. Public Health Reports, 113(4), 353-358.

Cruz, M., Katz, D., & Suarez, J. (2001). An assessment of the ability of routine restaurant inspections to predict food-borne outbreaks in Miami-Dade County, Florida. American Journal of Public Health, 91(5), 821-823.

Florida Department of Business and Professional Regulation, Division of Hotels and Restaurants. (2000). Food safety training certification program application. http://www.myflorida.com/dbpr/hr/forms/7000/form/dbpr_hr_7013_food_safety_training_certification_program_appli.pdf (27 Sep. 2004).

Food and Drug Administration, Center for Food Safety and Applied Nutrition. (2000). Report of the FDA retail food program database of foodborne illness risk factors. http://vm.cfsan.fda.gov/~dms/retrsk.html (9 Jan. 2004).

Irwin, K., Grendon, J., & Kobayashi, J. (1989). Results of routine restaurant inspections can predict outbreaks of foodborne illness: The Seattle-King County experience. American Journal of Public Health. 79(5), 586-590.

Mathias, R.G., Sitzo, R., Hazelwood, A., & Cocksedge, W. (1995). The effects of inspection frequency and food handler education on restaurant inspection violations. Revue Canadienne de Sante Publique, 86(1), 46-50.

McNamara, C. (1999). Basic guide to program evaluation. Management Assistance Program for Non-Profits, Free Management Library. http://www.mapnp.org/library/evaluatn/fnl_eval.htm (12 Jan. 2004).

Olsen, S. J., MacKinon, L. C., Goulding, J.S., Bean, N.H., & Slutsker, L. (2000). Surveillance for foodborne-disease outbreaks--United States, 1993-1997. Morbidity and Mortality Weekly Report (MWWR), 49(SS-1), 1-51.

Raval-Nelson, P., & Smith, P. (1999). Food safety certification and its impacts. Journal of Environmental Health, 61(7), 9-13.

Reed, L.E. (1982). Training effectiveness in school food service. Journal of the American Dietetic Association, 81(2), 176-178.

Riben, P.D., Mathias, R.G., Wiens M., Cocksedge, W., Hazelwood, A., Kirshner, B., & Pelton J. (1994). Routine restaurant inspections and education of food handlers: Recommendations based on critical appraisal of the literature and survey of Canadian jurisdictions on restaurant inspections and education of food handlers. Canadian Journal of Public Health, 85(Supp. 1), 567-570.

Roberta M. Hammond, Ph.D.

Robert G. Brooks, M.D.

Josefa Schlottmann, Ph.D.

David Johnson, M.D., M.S.

Rita J. Johnson, R.S.

TABLE 1 Number and Rate of Foodborne Outbreaks, by Contributing Factor,
Before and After Implementation of Training*

                                                        2001-2003 (After
Contributing Factor        1997-2000 (Before Training)  Training)
                                      Rate of
                           Number of  Outbreaks (per    Number of
                           Outbreaks  1,000)            Outbreaks

Poisonous substance         14         14                 4
accidentally added
Liquid/semisolid mix of     19         19                37
potentially hazardous
food
Raw/lightly cooked          39         39                47
Raw product contaminated    42         42                44
by animal/environment
Cross-contamination from   154        154               104
raw ingredient of animal
origin
Advance preparation         36         36                19
Bare-hand contact          126        126               151
Slow cooking                80         80                61
Insufficient time/          42         42                12
temperature cooking
Insufficient time/         123        123                35
temperature during hot-
holding
Inadequate cold-holding    229        229               178
temperature
Insufficient time/          37         37                19
temperature during
reheating
Infected food handler       46         46                42
Toxic tissue                24         24                20
Storage in contaminated     10         10                22
environment
Toxic container              2          2                 3
Polluted source             10         10                 1

                           2001-2003 (After
Contributing Factor        Training)          p-value**
                           Rate of Outbreaks
                           (per 1,000)

Poisonous substance          7                0.100
accidentally added
Liquid/semisolid mix of     64                <.001
potentially hazardous
food
Raw/lightly cooked          81                <.001
Raw product contaminated    76                 .002
by animal/environment
Cross-contamination from   179                 .904
raw ingredient of animal
origin
Advance preparation         33                 .634
Bare-hand contact          260                <.001
Slow cooking               105                 .046
Insufficient time/          21                 .012
temperature cooking
Insufficient time/          60                <.001
temperature during hot-
holding
Inadequate cold-holding    306                <.001
temperature
Insufficient time/          33                 .329
temperature during
reheating
Infected food handler       72                 .014
Toxic tissue                34                 .112
Storage in contaminated     38                <.001
environment
Toxic container              5                 .140
Polluted source              2                 .028

*A binomial test for difference of proportions was applied to each
factor to determine whether changes occurred in the rate of associated
outbreaks before (1997-2000) and after (2001-2003) training.
**The p-values indicate whether there was a significant change in the
rate of outbreaks associated with each contributing factor after
training began.

TABLE 2 Number and Rate of Individual Cases, by Contributing Factor,
Before and After Implementation of Training*

                                                       2001-2003 (After
Contributing Factor       1997-2000 (Before Training)  Training)
                          Number of                    Number
                          Cases      Rate (per 1,000)  Cases

Poisonous substance          35        6                  11
accidentally added
Liquid/semisolid mix of      84       15                 295
potentially hazardous
food
Raw/lightly cooked          107       19                 181
Raw product contaminated    374       66                 208
by animal/environment
Cross-contamination from  1,491      264                 492
raw ingredient of animal
origin
Advance preparation         304       54                 335
Bare-hand contact           950      168               1,214
Slow cooking                289       51                 341
Insufficient time/          271       48                  90
temperature cooking
Insufficient time/          594      105                 249
temperature during hot-
holding
Inadequate cold-holding     948      168                 929
temperature
Insufficient time/          224       40                 219
temperature during
reheating
Infected food handler     1,300      230                 470
Toxic tissue                 82       15                  86
Storage in contaminated      44        8                 147
environment
Toxic container               3        1                  11
Polluted source              37        7                   3

                          2001-2003 (After
Contributing Factor       Training)            p-value**
                          Rate of (per 1,000)

Poisonous substance         3                   .019
accidentally added
Liquid/semisolid mix of    82                  <.001
potentially hazardous
food
Raw/lightly cooked         51                  <.001
Raw product contaminated   58                  0.941
by animal/environment
Cross-contamination from  137                  <.001
raw ingredient of animal
origin
Advance preparation        94                  <.001
Bare-hand contact         339                  <.001
Slow cooking               95                  <.001
Insufficient time/         25                  <.001
temperature cooking
Insufficient time/         70                  <.001
temperature during hot-
holding
Inadequate cold-holding   259                  <.001
temperature
Insufficient time/         61                  <.001
temperature during
reheating
Infected food handler     131                  <.001
Toxic tissue               24                  <.001
Storage in contaminated    41                  <.001
environment
Toxic container             3                   .001
Polluted source             1                  <.001

*A binomial test for difference of proportions was applied to each
factor to determine whether changes occurred in the rate of associated
cases before (1997-2000) and after (2001-2003) training.
**The p-values indicate whether there was a significant change in the
rate of cases associated with each contributing factor after training
began.