Assessing the effectiveness of food worker training in Florida: opportunities and challenges.
From a public health standpoint, it is important to assess the effectiveness of food worker training programs in preventing foodborne outbreaks (McNamara, 1999). Food worker responsibilities include handling and preparing food that will be served to consumers, and most professionals agree that food worker training and education are essential in preventing foodborne disease by promoting good food safety and hygiene (Cotterchio, Gunn, Cofill, Tormey, & Barry, 1998; Riben et al., 1994).
Until recently in Florida, only food managers were mandated to be certified in food safety and sanitation. The assumption was that they would in turn train food workers employed in their establishments. In the year 2000, changes were made to Florida law requiring training of all food workers in food establishments inspected by the Florida Department of Business and Professional Regulation (DBPR). The change in training requirements, coupled with existing data on foodborne-disease outbreaks and cases compiled by the Florida Department of Health (FDOH), provided a unique opportunity to explore methods of assessing the effectiveness of food worker training requirements.
This paper discusses the challenges of measuring the effectiveness of food worker training, various methodologies that might be considered, and the results of utilizing trends in foodborne-outbreak contributing factors to evaluate Florida's recent change in food worker training requirements.
Background: Prior Efforts to Measure Food Worker Training Programs
Researchers have made numerous efforts to evaluate the effectiveness of food safety education (Mathias, Sizto, Hazlewood, & Cocksedge, 1995; Raval-Nelson & Smith, 1999; Reed, 1982). Most have focused on the effect of food worker training on critical inspection violations. A Canadian study concluded that although food-handling practices did improve and inspection scores were higher soon after training, no differences in scores were found during inspections performed six months after training (Mathias et al., 1995). Another study of the Food Safety Certification Program in Philadelphia indicated a positive impact on learned behaviors; however, the survey used to assess the effectiveness of certification training was too limited to determine the degree of application of the learned behaviors (Raval-Nelson et al., 1999).
In addition, several state and local health jurisdictions are looking at measuring the effectiveness of food worker training. The state of New York has collected data on factors contributing to foodborne illnesses for at least 10 years, but has not written any report or summary at this time. The Tacoma-Pierce County Health Department in Washington State has written a draft report on the subject using specially designed questionnaires to assess the extent to which knowledge about key food safety concepts was retained subsequent to training. The state of Minnesota is also conducting a study on food worker training.
Other colleagues responsible for investigating and preventing foodborne-disease outbreaks indicate an ongoing interest in assessing the effectiveness of food worker training. A presentation given at NEHA's Annual Educational Conference in Atlanta on July 1, 2001, was titled, "What Should Be Measured to Determine if a Food Safety Education Program Is Effective?" Furthermore, the Food and Drug Administration (FDA) has attempted to establish national baseline figures on the occurrence of foodborne-disease risk factors in its Report of the FDA Retail Food Program Database of Food borne Illness Risk Factors (FDA, 2000).
How best to evaluate the effectiveness of food worker training programs has not been established. Some studies have looked at utilizing the findings of restaurant inspections to predict foodborne-outbreak occurrence. These studies have reported mixed results. While the Washington study in Seattle-King County did find an association between outbreak occurrence and critical violations (among other factors) (Irwin, Grendon, & Kobayashi, 1989), a later study in Miami-Dade County, Florida, showed that inspections in that county did not reliably predict the occurrence of outbreaks (Cruz, Katz, & Suarez, 2001). The variety of prior approaches and varied results attest to the difficulty in measuring the effectiveness of food worker training.
Seven methods of assessing food worker training effectiveness and the limitations of each method were considered by the authors. Only the last method listed, Method 7, was utilized in the authors' attempt to assess the benefits of Florida's recent change in training requirements; however, the other methods may be appropriate and useful in other situations. All seven are listed here for review:
1. Comparison of the number of foodborne-illness outbreaks before and after implementation of food worker training. Limitation: Because outbreaks have a variety of causes and there is variability in outbreak reporting and detection, this approach lacks specificity.
2. Comparison of the number of cited critical violations in restaurants before and after implementation of food worker training. Limitation: Baseline data on violations before implementation of training were not available to the authors.
3. Correlation of the number of foodborne-illness outbreaks with the number of food workers trained after implementation of training. Limitations: food worker mobility and lack of consistent records on training of food workers statewide.
4. Comparison of the occurrence of outbreak contributing factors in food establishments inspected by DBPR (restaurants), where training is required, and the occurrence of these factors in other food establishments (e.g., schools, institutions and grocery store delicatessens), where the new 2000 food worker training is not required. Limitations: different workforce makeup in the various types of food establishments (restaurants generally having a less stable workforce than other food service establishments), differences in education and language capabilities of the workforce in various establishments, and differences in types of foods and food preparations required in different types of food establishments.
5. Comparison of trends in foodborne-illness outbreaks and cases associated with specific pathogens (e.g., hepatitis A, E. coli O157: H7, S. Typhi, Shigella spp., Salmonella spp., Norovirus, Staphylococcus) occurring before and after implementation of new training requirements. Limitations: Again, this measure is not specific enough to assess the impact of food worker education and involves many confounding factors.
6. Comparison of the training compliance rate of food establishments having outbreaks with the training compliance rate of food establishments not having outbreaks. Limitations: Current data regarding the identity of a restaurant involved in an outbreak are not available electronically, and manual abstraction is required.
7. Comparison of the trends, before and after implementation of training, in contributing factors associated with foodborne outbreaks and cases. The authors chose this method of analysis to assess the new 2000 worker training requirements in Florida and believe it offers the best comparison available at this time. The reasons this approach was chosen and its limitations will be covered in the discussion section of this paper.
Changes in Food Worker Training Requirements in Florida
Since 1988, it has been required that food service managers in DBPR-licensed food service establishments be trained in food safety standards (Florida Statutes, [section] 509.039). On January 1, 1998, further legislation was promulgated requiring the training of all public food service employees in professional hygiene and foodborne-disease prevention (Florida Statutes, Section 509.049). These training requirements delineated the areas of food safety and hygiene that must be covered in training. Food service establishments regulated by the Florida Department of Health (FDOH) and the Florida Department of Agriculture and Consumer Services are not covered by this requirement. DBPR establishments include over 40,000 Florida restaurants. While it was required that all food service workers be trained, it was assumed that the training would be provided by certified food managers working in the same establishment. No documentation or tracking of food worker training was required at that time.
In 2000, the Florida Legislature enacted legislation that further defines training requirements for all food workers in DBPR-licensed food service establishments (Florida Statutes, [section] 509.049(6)c). In the state of Florida, DBPR establishments include mostly restaurants and caterers, but not grocery stores, food processors, or institutional food service settings. These changes in the Florida Statutes require a food safety training certificate program for food service employees to be administered by a private, nonprofit provider chosen by DBPR's Division of Hotels and Restaurants. In addition, any food safety training programs established and administered to DBPR food workers before the effective date of the new requirements (July 1, 2000) had to be reviewed and approved by the Division of Hotels and Restaurants. By January 1, 2001, all currently employed DBPR food service workers had to receive food worker training by an approved certification program. Specifics of training requirements can be obtained from DBPR's Web site (Florida Department of Business and Professional Regulation, 2000).
In general, an approved certification program for food workers requires that employees be trained in 1) food safety criteria stated in Florida laws and rules, 2) major risk factors considered in foodborne outbreaks, and 3) the FDA Food Code's five intervention strategies. The Food Code's five intervention strategies focus on the following areas:
* demonstration of knowledge,
* employee health controls,
* controlling hands as a vehicle of contamination,
* time and temperature parameters for controlling pathogens, and
* consumer advisory.
Employees hired after November 1, 2000, had to receive certified training within 60 days of being hired. The revised law stated that certification is good for three years. After three years, an employee in a DBPR-licensed food service establishment must be retrained and recertified.
Available Data in Florida Regarding Food-borne-Outbreak Contributing Factors
FDOH has had a unique Food and Waterborne Disease Program in place since 1994, offering surveillance, investigation, and training assistance to the state's 67 county health departments. Data are collected from each of the 67 counties on a monthly basis by nine regional environmental epidemiologists employed by the DOH Bureau of Community Environmental Health. Foodborne-outbreak data are compiled on a quarterly basis to calculate Quarterly Performance Measures and yearly to produce an Annual Report on Food and Waterborne Diseases.
Since 1997, the Food and Waterborne Disease Program has been systematically collecting data on factors that contribute to foodborne-disease outbreaks. Contributing factors associated with foodborne outbreaks and cases that were consistently investigated and documented from 1997 through 2003 were used by the authors to evaluate the effectiveness of the new food worker training implemented in 2001.
The number of foodborne-disease outbreaks and cases associated with each of 17 contributing factors were obtained from annual reports produced by the Florida Department of Health. The 17 specific contributing factors chosen for comparison in this report are derived from the Center for Disease Control and Prevention's (CDC's) identification of five broad risk factors related to food employee behaviors and food preparation practices in retail food service establishments, and from the risk factors currently in use by the CDC Electronic Foodborne Outbreak Reporting System (CDC, 2003; Olsen, MacKinon, Goulding, Bean, & Slutsker, 2000). CDC's five broad risk factors are as follows:
* improper holding temperatures;
* inadequate cooking, such as undercooking of raw shell eggs;
* contaminated equipment;
* food from unsafe sources; and
* poor personal hygiene.
The numbers of foodborne outbreaks and cases associated with each of the 17 contributing factors were summarized into two categories: the four years (1997-2000) before implementation of food worker training and the three years (2001-2003) after implementation of training. (In the context of this discussion, the term "associated contributing factor" means that the contributing factor was documented to be present in the facility at the time of the outbreak.) A total of 1,001 outbreaks of foodborne disease were found to be associated with one or more of the 17 contributing factors in the four years before training implementation; 581 outbreaks were found to be associated with one or more of the contributing factors in the three years after training implementation. These totals were used to determine the rate of outbreaks associated with each contributing factor per 1,000 outbreaks. A total of 5,651 cases of foodborne disease were found to be associated with one or more of the contributing factors before training implementation, and 3,582 cases were found to be associated after training implementation. These totals were used to determine the rate of cases associated with each contributing factor per 1,000 cases.
Using the total number of foodborne-disease outbreaks or cases associated with each contributing factor per 1,000 outbreaks or cases, the authors analyzed the rates using a Z-test for the equality of two proportions (binomial distribution). This test for difference of proportions was applied to each contributing factor to determine whether a significant change in the rate of outbreaks or cases associated with that contributing factor occurred after training was implemented. The results of the analysis are summarized in Table 1 and Table 2.
Table 1 summarizes the number and rate of foodborne outbreaks, by contributing factors, before and after implementation of training in DBPR-inspected facilities. The p-values indicate whether there was a significant change in the rate of outbreaks associated with each contributing factor after training began. A statistically significant increase occurred in the proportion of outbreaks due to eight (47 percent) of the 17 contributing factors. These eight were as follows: liquid/semi-solid mix of potentially hazardous food, raw or lightly cooked food, raw product contaminated by animal or environment, bare-hand contact, slow cooking, inadequate cold-holding temperature, infected food handler, and storage in a contaminated environment. A significant decrease occurred in the proportion of outbreaks due to three (18 percent) of the 17 contributing factors. These three were as follows: insufficient time or temperature during cooking, insufficient time or temperature during hot-holding, and polluted source. In six categories (35 percent) there did not appear to be any statistically significant change.
The total number of foodborne-disease outbreaks in restaurants associated with one or more of the contributing factors that we studied decreased from 1,001 (in 1997-2000) to 581 (in 2001-2003). Before training, the average annual occurrence of foodborne-disease outbreaks associated with one or more of the contributing factors before training was 250 per year (in 1997-2000); after training it was 194 per year (in 2001-2003).
Table 2 summarizes the number and rate of individual cases, by contributing factor, before and after implementation of training in DBPR-inspected facilities. The p-values indicate whether there was a significant change in the rate of cases associated with each contributing factor after training began. In 2001-2003, a statistically significant increase was noted with respect to 10 (59 percent) of the 17 contributing-factor categories. A decrease was noticed in the proportion of cases caused by six (35 percent) of the contributing factors, and no change occurred in cases associated with one category (raw product contaminated by animal or environment).
The total number of foodborne-disease cases in restaurants associated with one or more of the contributing factors that the authors studied decreased from 5,651 cases (in 1997-2000) to 3,582 cases (in 2001-2003). Taking an average, the annual occurrence of foodborne-disease cases associated with one or more of the contributing factors before training was 1,413 per year (in 1997-2000); after training it was 1,194 per year (in 2001-2003).
It is generally accepted that proper education is a key to overcoming many adverse health outcomes, and this premise also applies to improving food safety for consumers. At the USDA-sponsored conference on food safety education in September 2002, attendees from 48 states and 10 countries were surveyed in several areas. Food worker training was considered the number-one priority with respect to food safety by the over 500 attendees surveyed.
Measuring the public health impact of food worker training is challenging, and many previous investigators have met with limited success. In this paper, the authors have explored the challenges of measuring the effectiveness of food worker training, as well as the various methodologies that might be used to assess the effectiveness of food worker training. After considering the various possible approaches, the authors chose to compare the occurrence of food worker contributing factors associated with foodborne outbreaks and cases of foodborne illness before implementation of training requirements on January 1, 2001, with the occurrence of contributing factors involved with outbreaks and cases occurring after January 1, 2001, in food service establishments licensed by the Florida DBPR.
Reasons for selecting contributing factors as a measure of the public health impact of food worker training include the following: 1) availability of these data in Florida, 2) the focus that food worker training places on these contributing factors, 3) the acceptance of the importance of these factors in foodborne outbreaks, and 4) lack of previous studies on this approach to evaluating training effectiveness.
The data in Table 1 and Table 2 summarize the findings of epidemiologic investigations subsequent to foodborne-disease outbreaks and cases and clearly indicate that food worker contributing factors are often documented to be involved in the occurrence of foodborne illness. The data from the authors' preliminary study and analysis illustrate that the relative incidence of many documented contributing factors has actually increased, while the incidence of others has decreased subsequent to the initiation of food worker training.
The overall annual occurrence of outbreaks and cases associated with one or more of the contributing factors the authors studied decreased after the implementation of the new training requirements in Florida. These data, however, have to be interpreted with caution, as outbreaks have a variety of causes and there is variability in reporting and detection of outbreaks and cases. Subsequent to the implementation of training, there were actually more outbreaks and cases involving multiple contributing factors. Again, there could be a variety of reasons for this outcome. For example, epidemiologists may have been more attuned or better trained to identify and document contributing factors in the later years of the study. In addition, data utilized in this analysis were derived from statewide data, and specific data were not available to indicate which food establishments were in compliance with training.
On the basis of the data available and the results of the authors' analysis, it is not possible at this time to make definite conclusions about the overall effectiveness of the newly implemented food worker training program. The data give information about the relative frequency of occurrence of contributing factors associated with outbreaks or cases. It does become apparent that following the trends of contributing factors associated with outbreaks may be a useful evaluation tool. Finding a negative trend in the incidence of contributing factors may be as useful as finding a positive trend, because both serve as guides for altering training material and methods and focusing training on needed areas. Utilizing contributing-factor data in conjunction with overall numbers of outbreaks and cases may provide additional useful information in the evaluation of training effectiveness.
More time is needed to assess whether this current approach is a useful method for tracking overall training results, or whether one of the other approaches described in this paper will be a better tool.
Regardless of the method chosen, there will be challenges in determining the success or failure of training. For example, restaurants have a special problem with maintaining an educated workforce because they have an extremely high turnover rate in comparison with other industries. While they are training all their staff within 60 days of hire, some establishments may also have to replace 75 percent of their staff within a few months' time. Training must start again with each new employee. This situation adds to the challenge of verifying the compliance rate of training programs in food establishments statewide.
In summary, common sense suggests that food worker training programs will improve food handler practices and decrease the risk of foodborne illness and foodborne outbreaks. Assessing the public health impact of food worker training will continue to be an important public health issue. A variety of challenges and limitations are involved in measurement of the overall effectiveness of food worker training programs. Following trends in the contributing factors associated with foodborne-diseases outbreaks and cases may be a useful approach for determining the effectiveness of food worker training, and it may be helpful in directing ongoing training efforts. Knowing the trends in contributing factors can help to determine areas of food worker safety that need emphasis in training programs. Further work needs to be done to establish the methods and approaches that are most useful for assessing the effectiveness and hence the public health impact of food worker training.
Corresponding Author: Roberta M. Hammond, Food and Waterborne Disease Coordinator, Florida Department of Health, 4052 Bald Cypress Way, Bin #A08, Bureau of Community Environmental Health, Tallahassee, FL 32399-1712. E-mail: email@example.com.
Centers for Disease Control and Prevention, Foodborne and Diarrheal Disease Branch. (2003). Investigation of a foodborne outbreak reporting form. http://www.cdc.gov/foodborneoutbreaks/report_f/reporting_form.doc (4 Oct. 2004).
Cotterchio, M., Gunn, J., Coffill, T., Tormey, P., & Barry, M.A. (1998). Effect of a manager training program on sanitary conditions in restaurants. Public Health Reports, 113(4), 353-358.
Cruz, M., Katz, D., & Suarez, J. (2001). An assessment of the ability of routine restaurant inspections to predict food-borne outbreaks in Miami-Dade County, Florida. American Journal of Public Health, 91(5), 821-823.
Florida Department of Business and Professional Regulation, Division of Hotels and Restaurants. (2000). Food safety training certification program application. http://www.myflorida.com/dbpr/hr/forms/7000/form/dbpr_hr_7013_food_safety_training_certification_program_appli.pdf (27 Sep. 2004).
Food and Drug Administration, Center for Food Safety and Applied Nutrition. (2000). Report of the FDA retail food program database of foodborne illness risk factors. http://vm.cfsan.fda.gov/~dms/retrsk.html (9 Jan. 2004).
Irwin, K., Grendon, J., & Kobayashi, J. (1989). Results of routine restaurant inspections can predict outbreaks of foodborne illness: The Seattle-King County experience. American Journal of Public Health. 79(5), 586-590.
Mathias, R.G., Sitzo, R., Hazelwood, A., & Cocksedge, W. (1995). The effects of inspection frequency and food handler education on restaurant inspection violations. Revue Canadienne de Sante Publique, 86(1), 46-50.
McNamara, C. (1999). Basic guide to program evaluation. Management Assistance Program for Non-Profits, Free Management Library. http://www.mapnp.org/library/evaluatn/fnl_eval.htm (12 Jan. 2004).
Olsen, S. J., MacKinon, L. C., Goulding, J.S., Bean, N.H., & Slutsker, L. (2000). Surveillance for foodborne-disease outbreaks--United States, 1993-1997. Morbidity and Mortality Weekly Report (MWWR), 49(SS-1), 1-51.
Raval-Nelson, P., & Smith, P. (1999). Food safety certification and its impacts. Journal of Environmental Health, 61(7), 9-13.
Reed, L.E. (1982). Training effectiveness in school food service. Journal of the American Dietetic Association, 81(2), 176-178.
Riben, P.D., Mathias, R.G., Wiens M., Cocksedge, W., Hazelwood, A., Kirshner, B., & Pelton J. (1994). Routine restaurant inspections and education of food handlers: Recommendations based on critical appraisal of the literature and survey of Canadian jurisdictions on restaurant inspections and education of food handlers. Canadian Journal of Public Health, 85(Supp. 1), 567-570.
Roberta M. Hammond, Ph.D.
Robert G. Brooks, M.D.
Josefa Schlottmann, Ph.D.
David Johnson, M.D., M.S.
Rita J. Johnson, R.S.
TABLE 1 Number and Rate of Foodborne Outbreaks, by Contributing Factor, Before and After Implementation of Training* 2001-2003 (After Contributing Factor 1997-2000 (Before Training) Training) Rate of Number of Outbreaks (per Number of Outbreaks 1,000) Outbreaks Poisonous substance 14 14 4 accidentally added Liquid/semisolid mix of 19 19 37 potentially hazardous food Raw/lightly cooked 39 39 47 Raw product contaminated 42 42 44 by animal/environment Cross-contamination from 154 154 104 raw ingredient of animal origin Advance preparation 36 36 19 Bare-hand contact 126 126 151 Slow cooking 80 80 61 Insufficient time/ 42 42 12 temperature cooking Insufficient time/ 123 123 35 temperature during hot- holding Inadequate cold-holding 229 229 178 temperature Insufficient time/ 37 37 19 temperature during reheating Infected food handler 46 46 42 Toxic tissue 24 24 20 Storage in contaminated 10 10 22 environment Toxic container 2 2 3 Polluted source 10 10 1 2001-2003 (After Contributing Factor Training) p-value** Rate of Outbreaks (per 1,000) Poisonous substance 7 0.100 accidentally added Liquid/semisolid mix of 64 <.001 potentially hazardous food Raw/lightly cooked 81 <.001 Raw product contaminated 76 .002 by animal/environment Cross-contamination from 179 .904 raw ingredient of animal origin Advance preparation 33 .634 Bare-hand contact 260 <.001 Slow cooking 105 .046 Insufficient time/ 21 .012 temperature cooking Insufficient time/ 60 <.001 temperature during hot- holding Inadequate cold-holding 306 <.001 temperature Insufficient time/ 33 .329 temperature during reheating Infected food handler 72 .014 Toxic tissue 34 .112 Storage in contaminated 38 <.001 environment Toxic container 5 .140 Polluted source 2 .028 *A binomial test for difference of proportions was applied to each factor to determine whether changes occurred in the rate of associated outbreaks before (1997-2000) and after (2001-2003) training. **The p-values indicate whether there was a significant change in the rate of outbreaks associated with each contributing factor after training began. TABLE 2 Number and Rate of Individual Cases, by Contributing Factor, Before and After Implementation of Training* 2001-2003 (After Contributing Factor 1997-2000 (Before Training) Training) Number of Number Cases Rate (per 1,000) Cases Poisonous substance 35 6 11 accidentally added Liquid/semisolid mix of 84 15 295 potentially hazardous food Raw/lightly cooked 107 19 181 Raw product contaminated 374 66 208 by animal/environment Cross-contamination from 1,491 264 492 raw ingredient of animal origin Advance preparation 304 54 335 Bare-hand contact 950 168 1,214 Slow cooking 289 51 341 Insufficient time/ 271 48 90 temperature cooking Insufficient time/ 594 105 249 temperature during hot- holding Inadequate cold-holding 948 168 929 temperature Insufficient time/ 224 40 219 temperature during reheating Infected food handler 1,300 230 470 Toxic tissue 82 15 86 Storage in contaminated 44 8 147 environment Toxic container 3 1 11 Polluted source 37 7 3 2001-2003 (After Contributing Factor Training) p-value** Rate of (per 1,000) Poisonous substance 3 .019 accidentally added Liquid/semisolid mix of 82 <.001 potentially hazardous food Raw/lightly cooked 51 <.001 Raw product contaminated 58 0.941 by animal/environment Cross-contamination from 137 <.001 raw ingredient of animal origin Advance preparation 94 <.001 Bare-hand contact 339 <.001 Slow cooking 95 <.001 Insufficient time/ 25 <.001 temperature cooking Insufficient time/ 70 <.001 temperature during hot- holding Inadequate cold-holding 259 <.001 temperature Insufficient time/ 61 <.001 temperature during reheating Infected food handler 131 <.001 Toxic tissue 24 <.001 Storage in contaminated 41 <.001 environment Toxic container 3 .001 Polluted source 1 <.001 *A binomial test for difference of proportions was applied to each factor to determine whether changes occurred in the rate of associated cases before (1997-2000) and after (2001-2003) training. **The p-values indicate whether there was a significant change in the rate of cases associated with each contributing factor after training began.
|Printer friendly Cite/link Email Feedback|
|Author:||Johnson, Rita J.|
|Publication:||Journal of Environmental Health|
|Date:||Oct 1, 2005|
|Previous Article:||Water availability and usage on the New Mexico/Mexico border.|
|Next Article:||Food safety issues and training methods for ready-to-eat foods in the grocery industry.|