Thursday, August 14th the City of Toronto experienced a wide spread power outage that encapsulated 50-60 million people throughout the Northeast United States and into large parts of Ontario in Canada.
During the blackout, many viewed this only as a minor inconvenience, however, as emergency responders, we look back on that day as more than "just a minor inconvenience."
The City of Toronto has recently completed a major upgrade and redesign of its public safety emergency radio communications system, which is shared by police, fire and EMS. Included in the redesign were all of the robustness and continuity considerations that one would expect. Uninterrupted power supply equipment, generators, transfer switches, fully redundant back-up communications centres, site linking redundancy and route diversity have all been included in the implementation. Lessons learned from the 1998 Ice Storm and Y2K preparations were also incorporated. At first glance, it would appear that the City and the system should have weathered a 24-36 hour power interruption easily and with absolute confidence.
Murphy's Law states that: "Anything that can go wrong will go wrong".
In spite of a solid system design, two of the four physical sites containing the critical system controllers suffered immediate electrical malfunctions. At one site, the generator failed to start, while at another site, it started but quickly experienced a severe and incapacitating malfunction which damaged most of the equipment connected to it, including the building's elevator.
The first generator was repaired within a few hours. Unfortunately, the situation was not the same for the generator at the second site. Too compensate for this, portable generators were dispatched to the site for interim use. As a consequence of the significant electrical malfunction, the building elevator was unavailable. Crews from two fire stations alternated in manually hauling portable generator up more than twenty floors to the building's roof. Police and Fire technical staff worked together at both locations to bring the systems back on line.
A specific review of maintenance records revealed that the generator that failed to start at the primary site had been tested one week prior and had performed as expected. Lesson learned - stuff happens. An analysis of the failure at the second site indicates that some of the problem might be attributable to a less than optimal maintenance process at that site. Unfortunately, it is one of few available sites offering adequate radio coverage in that area of the city.
Lesson learned: When a critical site is leased and out of direct control of the public safety agency, it may be difficult to enforce a rigorous inspection and maintenance program of the actual building systems.
-Manually carrying generators up to 20 story's and keeping them refueled requires a lot of effort
