Demonstrating Your Fire Prevention Program’s Worth – Part 2

Part one of this article discussed the reasons we incorporate evaluation as part of our programs and resources available for program evaluation. Part two will discuss the road to effective evaluation and the types of evaluation. As mentioned in the...


Enforcement – Enforcement is used to change negative behaviors displayed. This is generally accomplished through legislative actions and the creation/application of codes. These codes advise us what we need to do and are generally established to prevent negative behaviors that, in the past, have lead to fires or injury.

The three techniques above were the basis of risk reduction and fire prevention. They were often combined to produce positive results. Recently, two more areas of risk reduction have been added:

Economic Incentives/Disincentives – There are economic incentives/disincentives with positive impacts, such as the insurance industry offering discounts to property owners who protect their property with fire sprinkler systems. Negative impacts include fines or costs for failing to provide the minimum, a citation or court summons issued by a municipality.

Emergency Response – The introduction of emergency response is used to reduce, but not eliminate, community risk reduction. This area requires that the community provide a rapid response coupled with a trained and adequately staffed workforce. The ultimate goal is to have the five E’s working together to provide a successful community risk-reduction program.

Evaluation

Evaluation is often confused with research. While some of the same principles apply, the main difference, according to Michael Q. Patton, is “Research seeks to prove, evaluation seeks to improve.” Evaluation identifies areas that work and areas that need improvement. It shows the community their investment in the program pays dividends. It also helps to decide which program to invest in when we have to make a choice in tough economic times. There are four types of evaluation: Formative, Process, Impact and Outcome. We will now review the basics of each type of evaluation.

Formative evaluation is used during the planning stages of new program development or existing program revision. The process starts with an analysis of the target population. Next, we must determine there is actually a problem. This assures we do not waste time, talent or money. We may see an educational program that addresses a risk in another community, which may not be a risk within your community. The delivery method and materials utilized must be reviewed. Running a pilot program helps identify issues on the delivery. Participant feedback regarding the delivery or materials identifies improvements needed. The program suffers when the process is not completed.

Process evaluation shows the implementation stage of the program by documenting activities. The fire service excels in this area by documenting emergency responses, property loss or response times. This can show how active a program is, or measure the community’s response through attendance or track items/materials delivered.

Impact evaluation shows the short-term effects of the program. This demonstrates a change in knowledge or awareness by the participant. It is shown through measuring learning or observing actions. A common method to measure change in knowledge is a pre-test and post-test comparison. A common method of observing action is witnessing a school fire drill. Both evaluations need to be documented.

Outcome evaluation shows the long-term effects of the program. This measures the big picture change through reduction. This includes the reduction of property loss, firefighter injuries or civilian deaths as a result of the program/project. These long-term effects are generally reviewing information over a couple years. Benchmarks and trends are used to determine whether there was a positive outcome as a result of the program/project.

In the words of Mike Weller (Hagerstown MD, FD retired), “Evaluation will help us work smarter, not harder.” Implementing the evaluation phase will help improve problem areas and provide documentation of success. This will prevent us from wasting time, talent and dollars on ineffective programs. It will also help to identify areas of improvement when there are changes/outside influences that affect the program.

The series will conclude with a discussion of the stakeholder’s responsibilities, evaluation examples and performance measurements for the areas of public education, fire inspection/code enforcement, fire and life safety plan review and fire investigation.