FDA Logo U.S. Food and Drug AdministrationCenter for Food Safety and Applied Nutrition
 U.S. Department of Health and Human Services

FDA Home Page | CFSAN Home | Search/Subject Index | Q & A | Help

horizontal rule

April 17, 2003

horizontal rule

National Conference on Interstate Milk Shipments (NCIMS)

Evaluation of the NCIMS HACCP Pilot Program Phase II Expansion

Appendix III. Summary of Regulatory Information

(Inspections, HACCP Audits, State Ratings, Pre and Post HACCP Evaluations, FDA Check Ratings, Regulatory Samples)

(Table of Contents)

Comparison of Inspection, Audit and Sample Data from HACCP Pilot Plants

Summary:

In general, items marked on state regulatory inspections, state listings, pre and post-HACCP surveys, state HACCP audits, state HACCP listing audits, and FDA HACCP audits showed good consistency relative to the frequency and types of items debited. State listings and state listing audits generally reflect the conditions noted on state regulatory inspections and state audits.

Comparison of Pre-HACCP and Post-HACCP milk product and water sample results indicate that HACCP Pilot plants performed at least as well under the HACCP system as they did under the traditional system.

The majority of traditional inspections and HACCP audits debited requirements normally managed by prerequisite programs under HACCP. Both the traditional inspection system and HACCP audits concentrated their debits in this area.

Comparison of pre and post-HACCP surveys from HACCP pilot plants that participated in Phase I & II conducted using traditional rating methods and procedures showed:

  1. (1) Three out of four plants (where more complete data was available) improved or performed at the same level when compared to pre-HACCP results.
  2. (2) Items marked on the surveys were less consistent than traditional inspections and listings. This may be true because:
    1. (a) HACCP is a young system being implemented as opposed to the traditional system that has been in place for many years; or
    2. (b) There actually was an increased level of non-compliance with PMO "housekeeping" requirements under HACCP; or
    3. (c) The small sample size for the data may not reflect a true picture of conditions in the HACCP Pilot plants; or
    4. (d) Comparing marks from traditional inspections with audit findings is likecomparing oranges and apples, the two are not the same.

Audit findings indicate additional emphasis and training are necessary to ensure previous audit findings are corrected in a timely manner and the hazard analysis and hazard plan are properly prepared and complete.

Based on the comparison of inspection, rating, check rating and sample data from state records before the implementation of the HACCP pilot program with similar data during the HACCP pilot showed HACCP pilot plant compliance not to be significantly different from the same plant's compliance with the PMO under the traditional system.

Data included in study:

Inspection, listing, check-rating, sampling, auditing, pre-HACCP and post-HACCP survey data from HACCP listed plants was used to prepare this report. Data was requested for each plant two years prior to the plants participation in HACCP through December 31, 2002. No date after December 31, 2002 was included in this analysis. Data from plants that did not achieve HACCP listed status or that were HACCP listed in the final months of 2002 and did not have any sample or audit data after the implementation of the HACCP system in the plant prior to December 31, 2002 were also excluded from the analysis.

Analysis of traditional inspection, listing, and check-rating data from Table I

There were 100 state regulatory inspections, 13 state listings and 11 FDA check-ratings conducted under the traditional system that are included in the data.

  1. Comparison of state regulatory inspections, state listings, and FDA check-ratings data:

    As expected, the greatest number and variety of violations were marked on state regulatory inspections. Fewer violations were documented on state listings and check-ratings than were marked on state regulatory inspections.

    Similarities between items marked:

    The following items were marked on between 39% and 64% of all state regulatory inspections respectively:

    • Items 15aa - Protection from contamination;
    • 2a - Walls and ceilings, smooth, washable, good repair;
    • 9a - Milk plant cleanliness; and
    • 1a - Floors, smooth, impervious.

    The following items marked on more than 10% of state regulatory inspections also included 4 out of the 6 most frequently marked items on state listings:

    • 15aa - Protection from contamination
    • 9a - Milk plant cleanliness
    • 1a - Floors, smooth, impervious
    • 5a - Separate rooms

    The following items marked on more than 10% of state regulatory inspections also included 3 out of the 6 most frequently marked items on check-ratings:

    • 15aa - Protection from contamination
    • 5a - Separate rooms
    • 10a - Sanitary piping, smooth, impervious

    Differences between items marked:

    The following items were marked on 15% to 23% of state listings, but less than 10% of the time on state regulatory inspections:

    • 4a - Lighting and ventilation
    • 22a - Surroundings, neat and clean

    The following items were marked on 18% to 27% of check-ratings, but less than 10% of the time on state regulatory inspections:

    • 18a - Bottling and packaging performed in a plant where contents pasteurized
    • 19a - Capping and/or closing performed in sanitary manner
    • 18b - Bottling and packaging performed in a sanitary manner

    Conclusions:

    • In general, there is good consistency between items marked on state regulatory inspections when compared to state listings or to FDA check-ratings.
    • State regulatory inspections did not emphasize Item 4a concerning lighting and ventilation or Item 22a concerning surroundings as much as state listings.
    • State regulatory inspections did not emphasize Items 18a, 18b, and 19a related to packaging finished dairy products as much as FDA check-ratings.
  2. Comparison of state regulatory inspections, state listings, and FDA check-ratings with Pre-HACCP Baseline surveys:
    • Eight out of ten items marked on Pre-HACCP baseline surveys were included within the items marked more than 10% of the time on state regulatory inspections.
    • Three out of ten items marked on Pre-HACCP baseline surveys were included within the items marked more than 10% of the time on state listings and FDA check-ratings.
    • Items 15aa (protection from contamination) and 5a (separate rooms) showed the most consistency and were marked on between 15.4% and 100% of state regulatory inspections, state listings, FDA check-ratings, and Pre-HACCP surveys. Items 9a (milk plant cleanliness) and 10a (sanitary piping) were each marked between 15% and 40% of the time on state regulatory inspections, state listings, and the Pre-HACCP surveys.
    • Items 7a (water supply) and 16a1a (batch pasteurization) were marked more often on Pre-HACCP surveys than on state regulatory inspections, state listings, or FDA check-ratings.

    Conclusions:

    Considering the small number of Pre-HACCP surveys included in the data, items marked were generally consistent with debits marked on state regulatory inspections, state listings and FDA check-ratings. All items marked better than one half of the time on Pre-HACCP listings were also among the most debited items marked during state regulatory inspections, listings, and FDA check-ratings.

  3. Comparison of state regulatory inspections, state listings, and FDA check-ratings with Post-HACCP Baseline surveys:
    • Six out of eight items marked on Post-HACCP baseline surveys were included within the items marked more than 10% of the time on state regulatory inspections.
    • Three out of eight items marked on Post-HACCP baseline surveys were included within the items marked more than 10% of the time on state listings and FDA check-ratings.
    • Item 15aa (protection from contamination) showed the most consistency and was marked between 38% and 64% of state regulatory inspections, state listings, and FDA check-ratings.
    • Items 4b (lighting and ventilation) and 13a (storage of cleaned containers and equipment) were marked more often on Post-HACCP surveys than on state regulatory inspections, state listings, or FDA check-ratings.

    Conclusions:

    In general, items marked on state regulatory inspections, state listings, FDA check-ratings and Post-HACCP surveys showed good consistency relative to the frequency and types of items debited.

  4. Comparison of Pre-HACCP and Post-HACCP surveys:

    The following five out of ten items marked on Pre-HACCP surveys were also marked on Post-HACCP surveys:

    • 15aa - Protection from contamination;
    • 2a - Walls and ceilings;
    • 9a - Milk plant cleanliness;
    • 11a - Construction and repair of containers and equipment; and
    • 10a - Sanitary piping, smooth and in good repair.

    The following five out of Eight items marked on Post-HACCP surveys were also marked on Pre-HACCP surveys:

    • 15aa - Protection from contamination;
    • 2a - Walls and ceilings;
    • 9a - Milk plant cleanliness;
    • 11a - Construction and repair of containers and equipment; and
    • 10a - Sanitary piping, smooth and in good repair.

    Fifty percent of items marked on Pre-HACCP and Post-HACCP surveys were identical.

    The following items were each marked once on a Pre-HACCP or Post-HACCP survey:

    • 1a - Floors, smooth, good repair;
    • 3a - Door and windows;
    • 4b - Lighting and ventilation;
    • 5a - Separate rooms;
    • 7a - Water supply;
    • 8a - Hand-washing facilities;
    • 13a - Storage of cleaned containers and equipment; and
    • 16a1a - Batch pasteurization.

    Conclusions:

    In general, items marked on Pre-HACCP and Post-HACCP surveys were consistent and tended to emphasize the same things.

Analysis of HACCP audit data from Table 2

  1. Findings related to all three categories of audit data (regulatory, listing and FDA).

    There were 54 state regulatory audits, 17 state listing audits and 8 FDA audits included in the data.

    The following items were marked more often than ten percent of the time for all three audit types (regulatory, listing, and FDA):

    1. 9c - Prerequisite programs and practices monitored;
    2. 9a - Required prerequisite program written;
    3. 9a5 - Protection from adulteration with lubricants;
    4. 10h - Other items as noted;
    5. 5b - Monitoring procedures followed;
    6. 1b - Hazard analysis identifies all food safety hazards;
    7. 10d - Labeling compliance; and
    8. 8a - Required information included in record.

    The next most marked items were:

    1. 7b - Verification activities are conducted;
    2. 9a2 - Condition and cleanliness of food contact surfaces;
    3. 1a - Hazard analysis conducted and written for each type of product;
    4. 6g - Corrective actions documented;
    5. 9d - Prerequisite program monitoring performed;
    6. 10c - Drug residue program records complete;
    7. 8b - Information entered on record at time observed;
    8. 9e - Corrections performed in a timely manner; and
    9. 9g - Prerequisite Program monitoring records adequately reflect conditions.

    Eleven out of the twelve most marked items on state listing audits were also marked in the top twenty most frequently marked items on state regulatory audits.

    Fourteen out of seventeen items marked more than once on FDA audits were also marked in the top twenty most frequently marked items on state regulatory audits.

    Conclusions:

    Based on the high percentage of like marks on state regulatory audits, state listing audits and FDA audits there is good consistency between the different audit types.

  2. Comparison of state listing audits with FDA audits:

    9 out of 12 or 75% of all debits marked more than 10% of the time on state listing audits were also marked on FDA audits. Audit items marked identically included:

    1. 9c- Prerequisite programs and practices monitored;
    2. 9a- Required prerequisite program written;
    3. 12a - Previous audit findings corrected;
    4. 9a5 - Protection of food from adulteration with lubricants;
    5. 10h - Other items as noted;
    6. 5b - Monitoring procedures followed;
    7. 1b - Hazard analysis identifies all potential food safety hazards;
    8. 10d - Labeling compliance; and
    9. 8a - Required information included in the record.

    FDA audits marked the following items 25% of the time when the state listings did not:

    • 1a - Hazard analysis conducted for each kind of product;
    • 10c - Drug residue control program records complete;
    • 8b - Processing information entered on records at time observed; and
    • 1c - Hazard analysis reassessed after changes.

    State listing audits marked the following items 10% or more of the time when FDA audits did not:

    • 7b - Verification activities are conducted and comply;
    • 9a2 - Condition and cleanliness of equipment food contact surfaces; and
    • 7e - CCP monitoring records reviewed and document values within limits.

    Conclusions:

    • Audit items 9c, 9a, 12a, 9a5, 10h, 5b, 1b, 10d, and 8a indicate there is good agreement between state regulatory audits, state listing audits and FDA audits on general compliance with NCIMS HACCP program requirements.
    • Differences between items marked on FDA audits as compared to items marked on state listing audits indicate FDA audits placed a greater emphasis on: (i) reviewing each plants hazard analysis and comparing it to the products produced; and (ii) ensuring the hazard analysis is maintained up to date and reviewed after changes or at least annually.
    • Differences between items marked on state listing audits as compared to items marked on FDA audits indicate state listing personnel placed a greater emphasis on equipment cleaning, pasteurization records, and verification activities.
    • The consistent marking of audit "item 12a - Previous audit findings corrected" is likely to be associated with: (i) the failure of state regulators performing audits to consistently establish timelines for corrections when completing state regulatory audits (especially during phase I of the HACCP Pilot); (ii) the fact that only one state performed followup audits routinely; and (iii) confusion on the part of state regulators and industry participants concerning their roles and proper completion of the audit form.
    • The frequent marking of Item 1a -- Hazard analysis conducted and written for each kind or group of milk product; Item 1b -- Hazard analysis identifies all potential food safety hazards; and Item 1c -- Hazard analysis reassessed after changes or at least annually, indicates additional training to emphasis the proper completion and maintenance of the hazard analysis is needed.

TABLE 1
Data from 100 state regulatory inspections, 13 state listings and 11 FDA check-ratings under the traditional system prior to the implementation of HACCP in the Pilot Plants
% marked on state reg. insp. % marked on state listings % marked on FDA Check-Ratings % marked on pre-HACCP Baseline Surveys % marked on Post-HACCP surveys Inspection item Description
64 38.5 45.5 100.0 75.0 15aa Protection from contamination, Operations conducted and located so as to preclude contamination of milk, milk products, ingredients, containers, equipment, and utensils
44   9.1 25.0 25.0 2a Walls and Ceilings, Smooth, washable, light-colored, good repair
40 15.4 9.1 25.0 25.0 9a Milk Plant Cleanliness, Neat, clean, no evidence of insects or rodents, trash properly handled
39 23.1     25.0 1a Floors, Smooth, impervious, no pools, good repair, trapped drains
25 7.7 9.1 25.0 50.0 11a Construction and repair of containers and equipment, Smooth, impervious, corrosion-resistant, not-toxic, easily cleanable materials, good repair, accessible for inspection
20   9.1     17a Cooling of milk, Raw milk maintained at 45 F or less until processed
19   9.1     14a Storage of single-service articles, Received, stored and handled in a sanitary manner, paperboard containers not reused except as permitted
19 15.4 18.2 50.0   5a Separate Rooms, separate rooms as required, adequate size
18 7.7   25.0   3a Doors and Windows, All outer openings effectively protected against entry of flies and rodents
17         12a Cleaning and sanitizing of containers/equipment, containers, utensils, and equipment effectively cleaned
15   18.2 25.0 50.0 10a Sanitary Piping, Smooth, impervious, corrosion-resistant, non-toxic, easily cleanable materials, good repair, accessible for inspection
12         12b Cleaning and Sanitizing of containers and equipment, Mechanical cleaning requirements of Ordinance in Compliance, records complete
11   9.1 25.0   8a Hand-Washing Facilities, Located and equipped as required, clean and in good repair, improper facilities not used
10 15.4       4a Lighting and ventilation, adequate in all rooms
8 7.7 27.3     18a Bottling and Packaging, Performed in a plant where contents finally pasteurized
7       25.0 13a Storage of cleaned containers and equipment, stored to assure drainage and protected from contamination
6         16eb Recording Charts, HTST & HTST pasteurizer chards comply with applicable Ordinance requirements
6 7.7 18.2     19a Capping, Capping and/or closing performed in sanitary manner by approved mechanical equipment
6         16ea Recording charts, batch pasteurizer charts comply with applicable Ordinance requirements
5         12c Cleaning and sanitizing of containers and equipment, approved sanitizing process applied prior to use of product contact surfaces
4         17c Cooling of milk, Approved thermometer properly located in all refrigeration rooms and storage tanks
4 23.1       22a Surroundings, Neat and clean, free of poled water, harborages and breeding areas
4         20a Personnel cleanliness, Hands washed clean before performing plant functions, rewashed when contaminated
3   9.1 25.0   7a Water supply, constructed and operated in accordance with Ordinance
3         21a Vehicles, Vehicles clean, constructed to protect milk
3   18.2     18b Bottling and packaging, performed in a sanitary manner by approved mechanical equipment
2         17d Re-circulated cooling water from safe source and properly protected, complies with bacteriological standards
2         17b Pasteurized milk and milk products, except those to be cultured, cooled immediately to 45 F or less in approved equipment, all milk and milk products stored threat until delivered
2         15bb Cross connections, overflow, spilled and leaked products or ingredients discarded
2         10c Sanitary piping, Pasteurized products conducted in sanitary piping , except as permitted by Ordinance
2 7.7       7b Water supply, No direct or indirect connection between safe and unsafe water
2         6a Toilet facilities, complies with ordinance
1   9.1     15ab Protection from contamination, air and steam used to process products in compliance with Ordinance
1         10b Sanitary piping, Mechanically cleaned lines meet Ordinance specs
1         19b Capping, Imperfectly capped/closed products properly handled
1   9.1     16da Regenerative heating, Pasteurized or aseptic product in regenerator automatically under greater pressure than raw product in regenerator at all times
1         16c2a Aseptic Processing, Flow-diversion device complies with Ordinance requirements
1         15bc Cross Connections, No direct connections between milk or milk products and cleaning and/or sanitizing solutions
1         11c Construction and repair of containers and equipment, approved single-service articles, not reused
1         9b Milk plant cleanliness, no unnecessary equipment
1         6c Toilet facilities, clean, well-lighted and ventilated, proper facilities
1         5c Separate rooms, storage tanks properly vented
1       25.0 4b Lighting and ventilation, well ventilated to preclude odors and condensation, filtered air with pressure systems
1         3b Doors and windows, outer doors self-closing, screen doors open outward
    9.1     15ac Approved pesticides, safely used
  7.7       7c Water supply, Condensing water and vacuum water in compliance with Ordinance requirements.
      25.0   16a1a Pasteurization Batch: Indicating/recording thermometers comply with ordinance
          16c1a Aseptic Processing indicating/recording thermometers comply with Ordinance
    9.1     16b2d Pasteurization, HTST: Flow promoting devices comply with Ordinance requirements

TABLE 1A
State Regulatory Inspections data under the Traditional System
Plant ID Inspection Date 1a 2a 3a 3b 4a 4b 5a 5c 6a 6c 7a 7b 8a 9a 9b 10a 10b 10c 11a 11c 12a 12b 12c 13a 14a 15aa 15ab 15bb 15bc 16c2a 16da 16ea 16eb 17a 17b 17c 17d 18a 18b 19a 19b 20a 21a 22a
A 11/20/00                           X                       X               X                    
A 2/13/01         X                 X                       X             X                      
A 5/8/01         X                 X               X                                            
A 8/14/01                           X               X       X     X                              
B 1/21/98   X                                                                                    
B 4/23/98     X                                             X               X                    
B 7/30/98   X                       X                       X               X                    
B 10/23/98                           X                       X               X                    
B 1/28/99 X X                       X                       X                 X                  
B 4/29/99                                         X       X                                      
B 7/30/99                           X             X                           X             X    
B 10/14//9                                                   X                                    
C 7/20/98                                                   X                   X                
C 10/16/98 X   X                     X                       X                                    
C 1/15/99                                         X         X                                    
C 4/15/99 X                                       X         X                                    
C 7/14/99 X                                                 X                                    
C 10/14/99                                         X   X     X                                    
C 11/1/00 X X                 X     X                                                            
D 4/12/00   X                       X                     X                                      
D 7/6/00   X X                     X         X             X                                    
D 10/3/00 X X                 X               X             X                                    
D 1/25/01   X                       X                       X                                    
D 4/17/01 X X                       X X       X           X X                                    
D 7/17/01 X X                                 X           X X       X                            
E 12/16/98 X                                       X         X           X                        
E 3/24/99                                     X             X             X                      
E 6/24/99 X                                       X         X             X                      
F 1/12/00 X X     X   X           X X   X         X   X   X X               X                    
F 4/12/00 X X X       X   X       X X   X               X X X                       X   X        
F 7/24/00 X X X       X   X         X   X     X         X X X               X       X            
F 10/26/00 X X         X       X     X   X     X           X X               X               X    
F 1/18/01 X X X   X   X           X X   X     X     X     X X               X       X            
F 4/9/01 X X X       X           X X   X     X     X   X X X               X               X    
F 7/11/01 X X X       X           X X   X     X         X X X                       X            
F 10/29/01 X X     X   X             X   X                 X X               X                    
F 1/3/02 X X X X                   X   X     X         X X X               X               X    
F 4/26/02 X X     X   X           X X   X                 X X               X       X   X        
F 7/23/02 X X X   X   X           X X   X     X           X X               X       X           X
G 5/26/98                                                   X                                    
G 8/27/98                         X                                                              
G 11/18/98                                     X     X                                            
G 2/3/99                                                   a                                    
G 3/9/99                                     X                                                  
G 5/6/99                                             X     X                                    
G 8/23/99                           X             X         X                                    
G 11/30/99                                                                                        
G 12/13/99                                                                                        
G 2/28/00                                                   X                                    
H 10/9/97                               X                                         X              
H 1/16/98                                                                 X                      
H 4/6/98                                                                                        
H 7/8/98                                                                                        
H 10/26/98                                                   X                                    
H 1/13/19\                                     X                                 X                
H 4/27/19                                                                                        
H 8/3/199   X X                         X X   X X X         X   X                     X          
H 11/24/19 X X                   X   X   X     X   X       X X   X               X     X          
I 1/4/01 X X         X                                     X                                    
I 4/5/01 X                                 X                                                   X
I 7/11/01 X                                 X                                                    
I 10/9/01 X                 X                                                                    
J 11/12/97 X X                       X                       X               X       X            
J 2/13/98 X X         X           X               X   X     X                                    
J 5/1/98 X X X   X   X                             X     X X           X   X           X        
J 8/14/98 X X         X                                     X           X                        
J 11/13/98 X X X   X                           X     X       X           X   X                    
J 2/1/99 X X         X                             X       X               X                    
J 5/5/99 X X X                                   X   X   X X           X   X                    
J 8/12/99 X X         X                       X         X X             X   X           X        
J 11/16/99 X X         X             X                       X         X                 X        
K 1/19/01                               X                                                        
K 2/23/01                                                                                        
K 4/27/01                                                                       X                
K 6/7/01                                                                                        
K 8/17/01                       X                                                                
K 10/9/01                                           X                                            
L 4/21/00 X X                       X         X             X                                    
L 7/26/00 X   X                     X             X X       X                                    
L 10/13/00 X   X     X                             X X                       X                    
L 1/19/01                                         X     X   X                                    
L 4/30/01   X                                 X             X             X                      
L 7/18/01   X                       X                       X                                    
L 10/17/01 X X                                       X       X                                    
L 11/30/01                           X             X                                              
L 12/28/01                           X         X             X                                    
L 1/4/02   X     X     X           X                       X                                    
M 2/3/00   X         X                                     X                     X             X
M 5/4/00                           X                                     X                     X
M 8/15/00                         X X         X                                                  
M 11/29/00   X                       X                                                            
M 2/8/01                                                   X                                    
M 5/4/01                                                                                     X  
M 6/26/01                                                                             X   X   X  
M 9/12/01   X X                     X                       X                                 X  
M 12/19/01   X                                               X                                    
M 3/6/02                                     X             X X                                  
M 6/1/02   X X                   X           X                                                  
  # Marked 39 44 18 1 10 1 19 1 2 1 3 2 11 40 1 15 1 2 25 1 17 12 5 7 19 64 1 2 1 1 1 6 6 20 2 4 2 8 3 6 1 4 3 4
  % Marked 39.0 44.0 18.0 1.00 10.0 1.00 19.0 1.00 2.00 1.00 3.00 2.00 11.0 40.0 1.00 15.0 1.00 2.00 25.0 1.00 17.0 12.0 5.00 7.00 19.0 64.0 1.00 2.00 1.00 1.00 1.00 6.00 6.00 20.0 2.00 4.00 2.00 8.00 3.00 6.00 1.00 4.00 3.00 4.00

 

TABLE 1-B
Items Debited on State Listings Under the Traditional System (Prior to HACCP Pilot Participation)
Plant ID Code Date of Rating 1a 2a 3a 4a 5a 7a 7b 7c 8a 9a 10a 11a 14a 15aa 15ab 15ac 16b2d 16da 17a 18a 18b 19a 22a
A 8/8/00                                              
A 9/20/01         X                             X   X  
D 5/2/00                      </