key: cord-0864655-qn8yifcd
authors: Wang, Chongjian; Wei, Sheng; Xiang, Hao; Xu, Yihua; Han, Shenghong; Mkangara, Ommari Baaliy; Nie, Shaofa
title: Evaluating the effectiveness of an emergency preparedness training programme for public health staff in China
date: 2008-01-15
journal: Public Health
DOI: 10.1016/j.puhe.2007.08.006
sha: 6102dd48ba28756830876fe88d80c8a81bcc802e
doc_id: 864655
cord_uid: qn8yifcd

BACKGROUND: The severe acute respiratory syndrome (SARS) crisis of 2003 provided a new urgency in China in terms of preparing public health staff to respond effectively to public health emergencies. Although the Chinese Government has already carried out a series of emergency education and training programmes to improve public health staff's capability of emergency preparedness, it remains unclear if these training programmes are effective and feasible. The purpose of this research was to evaluate an emergency preparedness training programme and to develop a participatory training approach for emergency response. METHODS: Seventy-six public health staff completed the emergency preparedness training programme. The effectiveness of the training was evaluated by questionnaire before training, immediately after training and 12 months after training (follow-up). Additionally, semi-structured interviews were conducted throughout the training period. RESULTS: The emergency preparedness training improved the knowledge levels and increased attitudinal and behavioural intention scores for emergency preparedness (P<0.01). The results at follow-up showed that the knowledge levels and attitudinal/behavioural intention scores of participants decreased slightly (P>0.05) compared with levels immediately after training (P<0.01). However, there was a significant increase compared with before training (P<0.01). Moreover, more than 80% of participants reported that the training process and resources were scientific and feasible. CONCLUSIONS: The emergency preparedness training programme met its aims and objectives satisfactorily, and resulted in positive shifts in knowledge and attitudinal/behavioural intentions for public health staff. This suggests that this emergency training strategy was effective and feasible in improving the capability of emergency preparedness.

Summary Background: The severe acute respiratory syndrome (SARS) crisis of 2003 provided a new urgency in China in terms of preparing public health staff to respond effectively to public health emergencies. Although the Chinese Government has already carried out a series of emergency education and training programmes to improve public health staff's capability of emergency preparedness, it remains unclear if these training programmes are effective and feasible. The purpose of this research was to evaluate an emergency preparedness training programme and to develop a participatory training approach for emergency response. Methods: Seventy-six public health staff completed the emergency preparedness training programme. The effectiveness of the training was evaluated by questionnaire before training, immediately after training and 12 months after training (follow-up). Additionally, semi-structured interviews were conducted throughout the training period. Results: The emergency preparedness training improved the knowledge levels and increased attitudinal and behavioural intention scores for emergency preparedness (Po0.01). The results at follow-up showed that the knowledge levels and attitudinal/behavioural intention scores of participants decreased slightly (P40.05) compared with levels immediately after training (Po0.01). However, there was a significant increase compared with before training (Po0.01). Moreover, more than 80% of participants reported that the training process and resources were scientific and feasible. 

Since the 9/11 disaster, anthrax bioterrorism, 'mad cow' disease, severe acute respiratory syndrome (SARS) and avian influenza outbreaks, public health emergencies have become an important threat to communities worldwide. Managing these emergencies and the threats they pose is part of the longterm government development plans in many countries, and expensive resources are being invested into preventing and responding to public health emergencies. 1 In China, surge capacity is one of the most urgent problems regarding public health emergency response at the present time, 2 along with the lack of equipment and the low efficiency of the public health emergency information system. Recognizing this, the Chinese Government carried out a series of emergency preparedness education and training programmes to improve the capability of public health staff to respond to emergencies nationwide. However, it remains unclear if these training programmes are feasible and effective in improving emergency preparedness.

Investigation has revealed that the emergency response of the public health sector was insufficient, especially the emergency preparedness of public health staff, such as not knowing the emergency response protocols and management procedures, and how to collect and analyse the relevant data during the SARS outbreak. 2, 3 Therefore, in order to change the current situation and improve the capability of public health staff in China to respond to emergencies, a pilot study was developed and supported by the Ministry of Health of the People's Republic of China (MOH) and the World Health Organization (WHO). The study was carried out by Tongji Medical College in Hubei Province from 2004 to 2006. The training was completed in 2005, and the follow-up survey was conducted 12 months later.

Like any other successful health education programme, the emergency preparedness training programme should be subjected to a process of continuous monitoring, control, evaluation and, if needed, relevant modifications. [4] [5] [6] [7] [8] The comprehensive evaluation of an emergency training preparedness programme should include its various aspects (contents, aims and objectives, training resources, methods, effects and impact), and it should also answer questions about the efficiency and impact of training on the participants. [9] [10] [11] This study highlighted the procedures used in the evaluation of an emergency preparedness training programme, and focused on its most important aspects: training resources; training process; and effectiveness of training (before training, immediately after training and 12 months later).

Seventy-eight trainees from the Centers for Disease Control and Prevention (CDC) in 18 cities in Hubei participated in the emergency preparedness training programme in 2005. Two participants did not complete their training and were not part of the evaluation (n ¼ 76). Trainers were selected based on their expertise in the field of public health emergency response, related training programmes and their involvement in continuous consultations on health service programmes, both educational and promotional. Trainers came from the MOH, WHO, Chinese CDC, Health Department of Hubei Province, Fudan University, Wuhan University and Huazhong University of Science and Technology.

The aims and objectives of training were designed carefully by educational and training experts with an intimate knowledge of the public health emergency response plan and the training programme, in consultation with public health personnel who did not participate in the training. The training programme was based on the US CDC's emergency preparedness core competencies for all public health workers. [12] [13] [14] In brief, the training consisted of: (1) the definition of public health emergency; (2) the public health workers' role during emergencies; (3) the responsibilities of local, province and government agencies during emergencies; (4) the role of the CDC during emergencies; (5) the CDC emergency response chain of command; (6) emergency communication strategies and use of special equipment; (7) emergency response protocols; and (8) management procedures, including the management of necessary supplies and equipment. The training contents and objectives were subjected to continuous monitoring and evaluation throughout the training period.

Various training methods were used, including case studies, workshops, tutorials, seminars, group discussions, role playing, drilling and fieldwork. Formal lecturing was the least used method. The training centre was equipped with modern audiovisual aids designed for training purposes. As well as the training logistics, other facilities and general services, such as transportation and accommodation, were provided free of charge to the participants.

Individual basic information, knowledge levels, attitudes and behaviours regarding emergency preparedness were investigated by questionnaire, which was designed by experts in the field of training programmes and continuous consultation on emergency management. In order to assess the questionnaire, a pilot test was undertaken among other public health personnel who did not participate in the training, and modifications were made by experts based on the feedback.

Thirty questions assessed the participants' knowledge of public health emergency competencies, which consisted of basic public health science knowledge, emergency management knowledge and emergency analytical/assessment skills (10 questions). If the correct answer was given to these questions, the participant received one point, whereas incorrect answers received no points. Eight items were designed to assess the staff's attitudinal and behavioural intentions related to the 'eight core competencies for public health services'. 15 Each of the eight items asked respondents to rate their attitudinal and behavioural intentions, as well as the frequency of their use of each of the competencies. Responses were rated on an ordinal scale (1 ¼ very low, 2 ¼ low, 3 ¼ middle, 4 ¼ high, 5 ¼ very high). Participants completed the first measurement (pre-test, baseline) on the first day of training. The post-training measurement (post-test) was conducted at the end of the last day of training. For the follow-up test, the participants were mailed a copy of the survey, with a self-addressed return envelope, 12 months after the training had been completed.

The training process and resources were subjected to continuous monitoring and evaluation by semi-structured interviews. The inclusion of the trainees in the evaluation process was extremely helpful in updating and modifying the programme. The items addressed in the semi-structured interviews were as follows: (1) the scientific methods offered; (2) the technical material presented;

(3) the performance of the trainer; (4) the benefits derived by the participant; (5) the use of the audiovisual aids; (6) the strengths and weaknesses of the session; and (7) final critical comments and remarks. The forms were distributed at the end of each session to be completed anonymously by each participant. The forms were analysed immediately and the results were shown to the trainer who had conducted the session. If any defects were revealed, the necessary rectifications were made immediately. Evaluation of workshops and fieldwork was carried out in a similar fashion. Feedback of the results of the evaluation was given to the participants.

Most data were reported as scores. Frequency and confidence scores were derived for each domain by participants' responses to the frequency questions and the self-efficacy questions. Repeated-measures analysis of variance was used to test differences between pre-test, post-test and follow-up test. The data from semi-structured interviews were categorized independently by three authors using the triangulation method, and the individual results of the analysis were compared and discussed until consensus was reached. All results were expressed as mean7standard deviation. Data were analysed by one-way analysis of variance using Statistical Package for the Social Sciences for Windows, Version 12.0 (SPSS. Inc., Chicago, IL, USA).

Seventy-six of the study participants completed the entire training programme and represented public health staff from the CDC of 18 cities (97.44% response). Most respondents were male (n ¼ 57, 75%) and over half (n ¼ 42, 55.26%) had earned a bachelor's or master's degree, of which one-sixth possessed Masters of Public Health degrees. Additionally, most participants (n ¼ 62, 81.58%) had more than 5 years of experience as public health staff. Some trainees (n ¼ 50, 65.79%) had participated in inter-related training approximately 12-24 months previously. The results of reliability assessment showed that test-retest reliability and the internal consistency of questionnaires was accredited to some extent (test-retest reliability of pretraining ¼ 0.83, Cronbach's alpha 40.61). The results of related analysis indicated that the construct validity of the questionnaire was of high quality (related coefficient fluctuated between 0.35 and 0.79, Po0.05). 16, 17 Knowledge levels

The investigation revealed that knowledge levels of public health emergency preparedness were relatively low before training. After training, a significant increase in the mean knowledge scores was observed (pre-test: 19.7972.41; post-test: 24.4970.86; followup test: 24.2471.58) (Po0.01). Basic public health science knowledge and emergency management knowledge scores decreased slightly (P40.05), but the mean scores for emergency analytical/assessment skills were increased dramatically in the follow-up test compared with the post-test (Po0.01). Furthermore, there was a significant increase in overall knowledge scores between the follow-up test and the pre-test (Po0.01) (Fig. 1 ).

Descriptive statistics on attitudinal and behavioural intentions at pre-test, post-test and follow-up test are presented in Table 1 . As mentioned above, the responses ranged from high (5) to low (1) . The results showed that participants reported a significant improvement in their attitudinal and behavioural intentions in all eight core competencies in the post-test compared with the pre-test. Twelve months later, there were slight decreases in participants' attitudinal and behavioural intentions in some core competencies, but the mean score for emergency analytical/assessment skills was significantly increased compared with the post-test (4.35 vs 3.69), and mean scores for policy development/programme planning skills (2.94 vs 3.95) and financial planning and management skills (2.66 vs 3.47) were decreased compared with the posttest (Po0.05).

The results of the semi-structured interviews showed that most participants (n ¼ 73, 96.05%) thought that the training methods were excellent/very good, and the training contents were clear and easy to understand. The remaining participants (n ¼ 3, 3.95%) indicated that the training methods needed to be improved/further developed. However, all of the participants recognized that the training was innovative.

Analysis showed that 80.26% (n ¼ 61) of participants were satisfied with the trainers' performance, and 19.74% (n ¼ 15) of participants thought that the trainers' performance needed to improve. However, no participants indicated that resource personnel were incompetent. Additionally, most participants (n ¼ 72, 94.74%) were very satisfied with the venue, training logistics and services, and only four participants (n ¼ 4, 5.26%) thought that logistics and services needed improvement.

Continuous medical education and training is a process of updating knowledge, developing skills, bringing about attitudinal and behavioural changes, and improving the capability of participants to perform their tasks efficiently and effectively. 18 Effective training methods are key to the success of an emergency training programme. A number of studies have shown that the training methods recommended by the present study educators were effective because different participants learn by different training methods, and methods of active training are especially helpful for adult learning. [19] [20] [21] Formal lecturing was the least used training method because trainees do not participate actively in the learning process and the outcome is inferior to methods of active learning. The results of the evaluation suggested that up-todate training of public health staff should focus on the development of effective training methods, and interactive training methods may help to increase the quality of training and improve retention of knowledge through immediate reinforcement of learning. 22, 23 Furthermore, comprehensive evaluation and feedback about the training programme were of vital importance for the participants and trainers as it helped participants to identify their limitations while monitoring their performance during the training period. Also, trainers tended to improve their performance as they were aware that it was being monitored and evaluated. Feedback of the results of evaluation of the training sessions to the trainers was found to be helpful in rectifying the weaknesses of sessions. 24 In addition, the mean scores of emergency analytical/assessment skills increased rather than decreased by 12-month follow-up. This is similar to results found by Qureshi et al. 14 For this type of phenomenon, one must consider the experience of the public health staff at the end of 2005. Before the follow-up survey, the majority of trainees had participated in avian influenza emergency response activities, thus providing practice and increasing perceived relevance of the training. As such, this probably had a positive effect on the effectiveness of training. Nevertheless, the increased overall knowledge score and the positive change in attitudinal and behavioural parameters suggested that training programmes on emergency preparedness resulted in gaining knowledge and shifts in attitude and behaviour. 

This study had a few potential limitations. The analysis was limited to staff who were primarily engaged in disease monitoring and control, and epidemiological investigations in the CDC. In addition, evaluations were based on changes over time without the use of a horizontal comparison group. Thus, it was not possible to fully determine which changes were due to the emergency preparedness training programme and which were the result of other factors. These results, however, remained constant throughout, which provides support that these changes were due to the training programme.

The effectiveness of any educational training programme depends on its continuous monitoring and evaluation, which should include appropriate and varied methods. Moreover, trainers and trainees should be actively subjected to the process of monitoring and evaluation, which was helpful in monitoring their overall performance. Immediate feedback with results analysis of the continuous monitoring and evaluation should be available to those involved so that necessary improvements can be made. The results of the evaluation suggested that the emergency training strategy was effective and feasible in improving the capability of public health staff to respond to an emergency.

A preliminary framework to measure public health emergency response capacity

Evaluation of the performance of responding to public health emergency for the workforce in CDC in Hubei

Report of the system construction on disease control and prevention. Hubei

Measuring effectiveness of TQM training: an Indian study

Emergency preparedness: one community's response

The road map to preparedness: a competency-based approach to all-hazards emergency readiness training for the public health workforce

Primary health workers in northeast Brazil

Evaluation of health impact assessment workshop. Criteria for use in the evaluation of health impact assessments

Business and public health collaboration for emergency preparedness in Georgia: a case study

Responsive evaluation of competencybased public health preparedness training programs

Emergency preparedness training for public health nurses: a pilot study

Emergency preparedness core competencies for all public health workers

Public health worker competencies for emergency response

Council on Linkages between Academia and Public Health Practice. Core competencies for public health professionals

Psychometric considerations in evaluating health-related quality of life measures

Psychometric theory

Training objectives, transfer, validation and evaluation: a SriLankan study

What matters most? Predictors of student satisfaction in public health educational courses

Applying educational gaming to public health workforce emergency preparedness

Relative effectiveness of worker safety and health training methods

Emergency preparedness and bioterrorism response: development of an educational program for public health personnel

Evaluating health impact assessment

Evaluation study of the training programs for health personnel in Al-Qassim, Saudi Arabia

The authors thank all of the participants and trainers for their hard work, and all of the coordinators for their support and help. In addition, the authors would like to thank C.K. Lee for his critical reading of the manuscript.

Not required.

World Health Organization.

None declared.