Part one of this article discussed the reasons we incorporate evaluation as part of our programs and resources available for program evaluation. Part two will discuss the road to effective evaluation and the types of evaluation. As mentioned in the previous article, while the fire service has employed some evaluation techniques, we have not shown how a service has been effective or where it needs to be improved. National Fire Protection Association’s NFPA 1710 and 1720 went a long way in employing evaluation strategies in the areas of response time and staffing to look at the effectiveness of fireground operations. We now evaluate fires, staffing at those fires, apparatus response time, extent of fire damage and firefighter fireground injuries. The standard’s main emphasis is evaluation for life safety, firefighter safety and property-loss stabilization.
Effective evaluation leads to the development of performance measurements, program benchmarks and trend analysis. It is created from standards and through risk-reduction analysis. The creation of the Firefighter Near-Miss Reporting System is an example of effective evaluation. This program examines near-miss incidents looking at the situation as a whole and decisions that lead to the incident. The objective is to improve firefighter safety, not to place blame. Evaluation is about documenting successes and improving service delivery.
Risk Reduction
Benjamin Franklin coined the phrase “an ounce of prevention is worth a pound of cure.” This is even more valuable in today’s economy with staffing cuts and increased demand for services. This applies to the four areas of fire prevention: Fire and Life Safety Education, Fire Inspection/Code Enforcement, Plan Review and Fire Investigation. These areas are often first on the cutting block as their importance has not been proven. Count the number of personnel in your department delivering prevention services versus those providing emergency response services.
Hopefully, you are familiar with the five “E’s” of risk reduction: Education; Engineering/Technology; Enforcement; Economic Incentives/Disincentives; and Emergency Response.
Education – The foundation of risk reduction is education. The goals of education are to provide information that creates awareness. Through this awareness we hope to change one’s knowledge about fire and fire safety. The change in knowledge, accompanied by the increased awareness, will lead to a change in attitude and behavior of the individual. Without the increase of awareness, there will not be a permanent change in attitude and behavior and the individual will resort to old behaviors.
People do not do what we want for three simple reasons: they were not trained or educated; they misunderstand the message; they just do not want to follow instructions. The first two are solved through education; the third by corrective actions (i.e. discipline and/or enforcement). Often we assume people understand how to be fire safe, even though they had no formal training in school. Most of what they have learned is through experience and or what they see on television/movies. Some of these examples are false depictions of reality (movies that show flames, but little smoke so we may see the action or actors).
It’s important to educate instead of lecture. People need to understand rather than be threatened with injury or property damage. Most people have the “I never thought or believed it would happen to me” attitude; one we often hear when informing an owner there has been a fire at their property.
Engineering/Technology – Engineering/Technology is used when education alone does not change behavior. This element removes the human factor. We saw this with the introduction of devices such as the smoke alarm and carbon monoxide detector. These devices provided notification that replaced fire-safety practices. They did not necessarily prevent fires, but provided notification to alert occupants on how to react (extinguish or evacuate).
Enforcement – Enforcement is used to change negative behaviors displayed. This is generally accomplished through legislative actions and the creation/application of codes. These codes advise us what we need to do and are generally established to prevent negative behaviors that, in the past, have lead to fires or injury.
The three techniques above were the basis of risk reduction and fire prevention. They were often combined to produce positive results. Recently, two more areas of risk reduction have been added:
Economic Incentives/Disincentives – There are economic incentives/disincentives with positive impacts, such as the insurance industry offering discounts to property owners who protect their property with fire sprinkler systems. Negative impacts include fines or costs for failing to provide the minimum, a citation or court summons issued by a municipality.
Emergency Response – The introduction of emergency response is used to reduce, but not eliminate, community risk reduction. This area requires that the community provide a rapid response coupled with a trained and adequately staffed workforce. The ultimate goal is to have the five E’s working together to provide a successful community risk-reduction program.
Evaluation
Evaluation is often confused with research. While some of the same principles apply, the main difference, according to Michael Q. Patton, is “Research seeks to prove, evaluation seeks to improve.” Evaluation identifies areas that work and areas that need improvement. It shows the community their investment in the program pays dividends. It also helps to decide which program to invest in when we have to make a choice in tough economic times. There are four types of evaluation: Formative, Process, Impact and Outcome. We will now review the basics of each type of evaluation.
Formative evaluation is used during the planning stages of new program development or existing program revision. The process starts with an analysis of the target population. Next, we must determine there is actually a problem. This assures we do not waste time, talent or money. We may see an educational program that addresses a risk in another community, which may not be a risk within your community. The delivery method and materials utilized must be reviewed. Running a pilot program helps identify issues on the delivery. Participant feedback regarding the delivery or materials identifies improvements needed. The program suffers when the process is not completed.
Process evaluation shows the implementation stage of the program by documenting activities. The fire service excels in this area by documenting emergency responses, property loss or response times. This can show how active a program is, or measure the community’s response through attendance or track items/materials delivered.
Impact evaluation shows the short-term effects of the program. This demonstrates a change in knowledge or awareness by the participant. It is shown through measuring learning or observing actions. A common method to measure change in knowledge is a pre-test and post-test comparison. A common method of observing action is witnessing a school fire drill. Both evaluations need to be documented.
Outcome evaluation shows the long-term effects of the program. This measures the big picture change through reduction. This includes the reduction of property loss, firefighter injuries or civilian deaths as a result of the program/project. These long-term effects are generally reviewing information over a couple years. Benchmarks and trends are used to determine whether there was a positive outcome as a result of the program/project.
In the words of Mike Weller (Hagerstown MD, FD retired), “Evaluation will help us work smarter, not harder.” Implementing the evaluation phase will help improve problem areas and provide documentation of success. This will prevent us from wasting time, talent and dollars on ineffective programs. It will also help to identify areas of improvement when there are changes/outside influences that affect the program.
The series will conclude with a discussion of the stakeholder’s responsibilities, evaluation examples and performance measurements for the areas of public education, fire inspection/code enforcement, fire and life safety plan review and fire investigation.
MARTIN M. KING serves as assistant chief in charge of the fire prevention bureau for the West Allis, WI, Fire Department. He has been with the department for 24 years and a member of the fire service for 32 years. As an adjunct instructor for the National Fire Academy, he instructs numerous courses in risk management and he has served on two recent fire prevention manual material review committees for IFSTA. He is currently a member of the Vision 20/20 committee working on performance measurements for fire prevention and education programs. Martin is a contributing expert to the Home Safety Council's Expert Network Academy. He has a bachelor's degree in business administration and is a recent graduate of the National Fire Academy's Executive Fire Officer Program.
