The year 2020 will be remembered for many things. COVID-19. Hurricanes. Wildfires. Murder Hornets. Multiple outages of our 9-1-1 national emergency number. While the first four calamities received a great deal of media attention, the last largely flew under the radar. Except for the millions of Americans who dialed 9-1-1 and got a recording. Sometimes, they didn’t even get that. Their call for help was answered, or more precisely unanswered, by silence or a busy signal.

The number nine-one-one is associated with reliable service, with numerous checks and balances to ensure maximum uptime. After all, what component of our response process is more critical? Still and all, last year, numerous outages were reported. Some were broad and long lasting. Others were localized and fleeting. All had the potential for tragedy.

The Federal Communications Commission (FCC) maintains a Network Outage Reporting System that specifies parameters under which carriers must report downtime. For example, conventional and wireless providers must advise communications centers of disruptions to service of at least “900,000 user minutes.” This is calculated by multiplying the duration of downtime by the number of affected users, so larger jurisdictions will have considerably tighter windows than sparsely populated rural areas. There are slightly different rules for Voice over Internet Providers (VoIP), and exceptions for disasters such as hurricanes, but by and large if the telephone service goes down for a measurable period, public safety must be notified. Many states also add requirements for carriers above and beyond federal regulations.

The 2020 Calendar

The FCC was unable to provide an exact number of downtimes for 2020, but based on those that did hit the media, there were sufficient incidents to cause concern. Even those that didn’t make the news are still troubling, especially if you’re the one trying to get help. Here is a partial listing of those outages that garnered coverage:

·       January               Several Michigan 9-1-1 centers go offline for three hours.

·        February             Centers in Utah reported difficulties with some carrier’s connections to 9-1-1.

·        March                  A California community has issues with conventional telephones reaching 9-1-1.

·        April                     An exchange specific disruption occurred in Colorado

·        May                      Saw a partial outage in an Illinois community.

·        June                     9-1-1 service in Arizona, Delaware, Illinois, Indiana, Minnesota, Nevada and                                      Ohio was compromised for more than an hour.

·        July                       A planned outage in a North Carolina community took down 9-1-1 service.                                              Residents were advised to report emergencies directly to the fire station.

·        August                 A California community lost wireless and internet access on two separate                                            occasions. 9-1-1 service was also impacted.

·        September         An outage lasting more than an hour hit fourteen states.

·        October               Six counties and Cherokee Tribal lands in North Carolina lost service due to a                                             fiber cut. In the same month, a widespread outage in Washington State lasted                                        about 30 minutes.

·        November          9-1-1 service in southern Kansas is interrupted.

·        December           A dozen 9-1-1 centers in Maine experience a 15 minute outage.

How 9-1-1 Works                                                                                   

In order to understand why 9-1-1 fails, it is important to understand how it functions when it is working properly. Public Safety Answering Points (PSAPs) receive ANI (Automatic Number Identification) and ALI (Automatic Location Identification) attached to conventional emergency calls. Consider these a robust form of caller ID. In the case of land line telephony, this data helps to direct the call to the proper PSAP through the use of a device known as a selective router. Wireless calls work slightly differently, relying on GPS and cell tower triangulation rather than a hard-wired address to direct delivery.

This information is presented to the call taker simultaneously with their picking up the receiver. For cell phones this is the approximate latitude/longitude of the caller or closest tower instead of a street address.

Legacy systems worked reliably because of their simplicity. Phones were at fixed locations. Historically, aside from a few systems which kept data locally, telephone companies maintained regional primary and secondary repositories for ALI. If one failed, the other picked up the load. Now, however, some 9-1-1 data, distribution, and even processing is consolidated on a national level. The addition of digital components such as wireless, VoIP, and text to the analog 9-1-1 backbone severely stretched its capacity. In fact, these “additions” now initiate more traffic than that generated by the original devices. Rather than having a seamless and purpose-built system, we have a variety of components that are forced to fit.

Why it Sometimes Doesn’t

Although it should be noted that the nationwide system remains relatively reliable, the number of vendors and networks involved in call delivery can sometimes complicate troubleshooting.

One such provider, Intrado, has been at the center of at least two separate major events. The multi-state September outage was traced back to a piece of equipment that was not forwarding calls properly. As a reference, they were previously involved in an April 2014 failure that blocked delivery of six thousand 9-1-1 calls. This was attributed to improper programming values.

The mix of wired, wireless, text, and VoIP technologies also opens up a wide variety of technical issues. Besides the potential for mechanical failure, there is the ever-present specter of human error. Loading test software instead of operational programs, unplugging a critical component, entering the wrong values, and severing critical circuits through incautious use of a backhoe are all listed as reasons the emergency number has failed.

Christmas Day, Nashville

But the capstone to 9-1-1 failures in 2020 was far from accidental. The Christmas day bombing of an AT&T facility in Nashville, Tennessee affected conventional, wireless, data and emergency number service in multiple states. Communications centers in Tennessee, Kentucky, and Alabama reported loss or restriction of service. While many emergency services were reportedly restored by next evening, some 9-1-1 lines were still down four days later. The results of the blast sent shockwaves through the public safety community. According to an AT&T spokesperson, “The Christmas morning explosion caused significant physical damage to a regional switching and core network hub that handles voice, data and video traffic across our wireless and fiber networks… Thanks to the heroic efforts of our team and the cooperation of federal, state and local law enforcement, we restored most service within 48 hours of the event.” While some praised AT&Ts efforts, a number of Tennessee 9-1-1 directors were less complimentary, feeling that answers were not quick in coming. The explosion was such that a state of civil emergency was declared in the city, television and internet were disrupted, and communications at Nashville Airport were compromised, grounding flights.

Collateral Damage

Perhaps as vexing was the impact upon FirstNet, the nationwide public safety communications network relied upon by Nashville first responders. Since AT&T provides both FirstNet and 9-1-1, assets of each were located in the building that was targeted by the blast. A FirstNet release advised: “Based upon initial analysis, it appears the FirstNet network infrastructure was not directly impacted by the explosion, and service continued operating on temporary battery power in the hours immediately following the event. However, because the bomb destroyed two local water mains, backup power generators were flooded and inoperable, and there was insufficient time to reroute all services before backup batteries were exhausted.”  Reports from public safety advise that a number of personal devices were put into play in order to deal with this emergency and others around the city.

Lessons Learned

Some uncomfortable truths have come from the Nashville bombing. Although no system can be expected to achieve 100% perfection, backups are supposed to exist to greatly reduce the potential danger. In the fire service, we are acutely aware of this need. It is part of our DNA, and we practice it daily through the use of two in/two out, backup lines, Rapid Intervention Teams (RIT), and the move-up of companies to cover vacant stations. It is not unreasonable that we should expect even greater redundancies from a system that provides the nationwide gateway between the public and public safety, and from another that promises nationwide interoperability for use in disasters.

A cornerstone of 9-1-1 is that there should never be a single point of failure. To this end, local communications centers go so far as to route cables through opposing corners of their buildings to guard against accidental disruption, and prohibit disparate circuits from sharing the same pole or manhole. Yet, here, a major – yet still singular – event debilitated not one, but numerous critical systems across a broad area. If Nashville is indicative of the networks nationwide, then our vulnerability is far greater than advertised.

Although a terrific explosion was involved, this does not negate the fact that the protections assumed to be provided by complex networking just weren’t there. A coordinated 9/11 style attack using vehicles instead of airplanes could cripple emergency reporting and response over vast portions of the United States. This is obviously unacceptable.

The Follow Up

In a New Year’s Day release, FirstNet Authority Board Chair Tip Osterthaler said: “The purpose of FirstNet is to provide connectivity to first responders regardless of conditions or circumstances, and that places a heavy burden on the Authority and AT&T, the service provider, to deliver the most resilient network achievable. As we absorb the lessons learned from this attack, we will adjust our risk management and investment strategies as appropriate to deal with the changing threat environment.”

The City of Nashville is considering the creation of a special commission, similar to that which reported on 9/11, to examine all aspects of the incident and response. The Tennessee Emergency Communications Board, a state agency chartered to “facilitate the technical, financial and operational advancement of Tennessee’s 911 systems” is conducting their own review. A January emergency meeting called on this topic was canceled, with discussion deferred until the regular meeting February 3^rd. While no physical AT&T presence was at the meeting, a letter from AT&T was read. According to reports, it shed no light on the lack of expected redundancy and its role in the failure. Two ETSB members have been authorized to work with AT&T moving forward. The next board meeting is scheduled for May.

Conclusions

As is the case with any complex investigation, it will take time to uncover all of the details of how a singular event caused so much damage to what were assumed to be robust systems, and more time yet to correct these vulnerabilities. One thing, however, is clear. While we consider our resources to be protected by distribution, this proves to be a false hope when hubs such as the one in Nashville serve as gathering points for disparate critical networks.

In the interim, the fire service must work with technicians and providers alike to assure that those redundancies we assumed to exist can be demonstrated as real. At no time should citizens be deprived of access to 9-1-1 as the result of a wanton act. We cannot afford another Nashville.