Clickability tracking pixel

Audit Warned of Outage Risk Before Kansas 911 System Failure

A roughly three-hour disruption sent police and sheriffs scrambling to post alternative phone numbers to social media, while multiple counties were unable to log into the state's 911 system during that time.

by Jonathan Shorman and Chance Swaim, The Kansas City Star / December 1, 2020
Shutterstock/Strike Pattern

(TNS) — Children are taught from a young age to dial 911 in an emergency. Whether it's a heart attack, a break-in or a fire, 911 answers the call right away.

A widespread 911 service outage across Kansas on Sunday afternoon is testing that assumption.

A roughly three-hour disruption sent police and sheriffs scrambling to post alternative phone numbers to social media. Multiple counties were unable to log into the state's 911 system during that time. What happened if you called varied from area to area, with some agencies able to reroute calls before service was fully restored by around 4:40 p.m.

The disruption — which encompassed the southern half of the state — is at least the third major failure of Kansas's 911 system in four years and comes after a 2018 audit warned that the system's design hadn't eliminated the risk of outages affecting multiple emergency departments.

Kansas has branded itself a national leader in building a state-of-the-art 911 system. Public safety officials have touted a years-long march toward implementing a statewide network that better shares critical data and responds to disasters.

But Sunday's outage is causing alarm among officials that such a large failure could have occurred — in the middle of a public health crisis, no less.

Rep.  John Carmichael , a Wichita Democrat who sits on the Kansas 911 system's governing body — the 911 Coordinating Council — said he wasn't surprised by the outage.

"The basic function of making sure we have a redundant system has been ignored and yesterday's events are just one more exemplification of this," Carmichael said. "We are teetering on the edge at some point of having a similar failure in the midst of, for example, a tornado or other significant statewide emergency."

The 911 Council, established by state law, has been guiding efforts for years to move Kansas to the next generation of service, commonly called NG911. That has involved everything from allowing residents to text 911 to ensuring calls can be quickly transferred between dispatch centers in case of a problem.

But the widespread outage on Sunday exposed a potential point of failure in a system that is supposed to withstand disasters of all kinds. The crisis began at about 1:35 p.m. when emergency agencies began experiencing call problems.

In Sedgwick County, home to Wichita, 911 staff were able to see emergency calls as they came in, but they couldn't answer.

"Staff attempted to reach all calls by making phone calls from their desk/administrative lines," Sedgwick County Emergency Communications Director  Elora Forshee  said.

While backups are in place, Forshee said "several layers failed yesterday."

Michele Abbott , who sits on the 911 Council's executive committee, said Sedgwick County was unable to log-on to the statewide 911 system on Sunday afternoon.

"We just really need to do a deeper dive, root-cause analysis report to see exactly what happened and why it happened," she said. "We had some counties not able to log on, but we don't know that their calls weren't routed somewhere else or that they weren't partially at the very least being properly answered somewhere else."

Wichita Police Department spokesman Officer  Paul Cruz  said the state's largest police department did its best to continue normal operations. He said field supervisors had officers dispatch from patrol substations, and officers were able to communicate through technology already available to the department, including laptops, police radios and the computer-aided dispatch system.

"The effects were minimal, so we were able to dispatch our officers to multiple calls," Cruz said.

It's unclear how many emergency calls didn't go through. "We do not know how many calls this impacted," Sedgwick County spokesperson  Kate Flavin  said. During a similar time period last Sunday, Sedgwick County 911 received 132 emergency calls, she said.

In a Facebook post, the 911 Council identified the problem as a "software conflict." The post also called the situation an "outage," but the word was later edited out.

Abbott on Monday repeated the explanation, but said the disruption didn't appear to be the result of a cyberattack.

AT&T is a major vendor for the state's 911 system. On Monday, a company spokesperson said that service was restored after a brief technical issue.

Audit warned of outages

Kansas emergency officials have been warned before about the potential for 911 problems—and at least two episodes in multiple years have demonstrated the system's vulnerabilities.

A 2018 audit commissioned by the Legislature's auditing operation, called the Division of Post Audit, found that 62 percent of emergency dispatch centers reported experiencing system down time. The audit also detailed two major outages that disrupted 911 service in dozens of counties.

"Based on survey responses and council interviews it was apparent large outage events have impacted the statewide platform," the audit report said.

On Jan. 27, 2018, a farmer placing fence post near Manhattan cut a fiber that ultimately affected 28 dispatch centers because of other maintenance problems that occurred around the same time. The audit noted that calls were rerouted to administrative lines during repairs.

On June 30, 2017, a problem during equipment maintenance also affected 28 centers. The audit said calls were successfully rerouted during the disruption, which lasted three and a half hours, but that eight centers experienced issues while using backup systems.

"Though corrective action has been taken to mitigate either event from happening again it should also be noted that system design does not eliminate the risk of future outages impacting multiple (centers)," the report said.

The audit report all but warned Kansas to expect future problems. In a "perfect world," the report said, 911 systems should also have just 5 minutes of downtime per year, a standard known as 5-9.

"With the Kansas state platform on a single network the system should not be expected to perform at a 5 nine standard," the report said.

Still, Abbott said Monday the Kansas 911 system's target is 5-9. Asked if an interconnected system creates concerns about vulnerabilities from a single failure, Abbott called it a "good question."

"Every single time there is an issue, we create redundant means to correct that moving forward," Abbott said.

Brandon Abley , technical issues director at the National Emergency Number Association, said old 911 systems are far more vulnerable than next generation systems. But much of the country is still transitioning to next generation systems, a position that leaves systems the most susceptible to problems.

"You're vulnerable to both kinds of failures and you're vulnerable to special failures," Abley said.

'Real consequences'

Sunday's failure has led to expressions of concern from state and local officials, who say the problem, which still isn't exactly clear, must be prevented from reoccurring.

Gov.  Laura Kelly  told reporters she was "obviously concerned something like that could happen." She said the Kansas Division of Emergency Management is working with local officials to identify the cause and ensure it doesn't happen again.

Sedgwick County Commissioner  David Dennis  said having a reliable, and always working, 911 line is "critical" to the Wichita area. That's why the state's 911 Coordinating Council was originally created, he said.

"That's the whole purpose of this large coordinating council is if somebody goes down, others can pick it up," Dennis said. "But I don't know if that was possible because it was an AT&T problem; it wasn't our problem, so I don't know if we were able to transfer our 911 functions to someone else."

Wichita Mayor  Brandon Whipple  said the city, county and state should work together to make sure an outage doesn't happen again — and need a strategy for what to do if it does.

"Our attitude shouldn't be that this is so rare that we shouldn't worry about it, but instead we should acknowledge that there's real consequences when it does happen, even if it's rare, so we have a better plan in place," Whipple said.

(c)2020 The Kansas City Star (Kansas City, Mo.). Distributed by Tribune Content Agency, LLC.

Looking for the latest gov tech news as it happens? Subscribe to GT newsletters.

E.REPUBLIC Platforms & Programs