IT leaders who’ve been at the center of response and recovery efforts reminisce on what went wrong and right, what works and what doesn’t, and how their experiences might be useful for others.
Katrina. Sandy. The Boston Marathon bombing. 9/11.
Some emergencies are indelible on the national consciousness to the point that they need no introduction. These tragedies are looked back upon as touchstones in America’s quest to be as prepared as possible for the next disaster.
Perfection is an impossible goal, but the public officials who’ve been at the center of response and recovery efforts can’t help but reminisce on what went wrong and right, what works and what doesn’t, and how their experiences might be useful for others forced to live through a similar event.
The people who manage the public’s technology and computer systems often are unheralded. But in an increasingly interconnected and online world, the lessons they’ve learned will only loom larger when — not if — the next large-scale disaster strikes.
Government Technology talked with officials who’ve worked amid some of America’s biggest disasters and emergencies of recent years. They offer practical, real-world perspective that we all can learn from.
A day before the official start of this year’s Atlantic hurricane season, Rafael Mena was invited to the White House. On May 30, shortly after embattled Veterans Affairs Secretary Eric Shinseki resigned, Mena sat across a table from President Obama.
Mena and two others, a Virginia official and the U.S. Energy secretary, were there to discuss innovative approaches to disaster preparedness. As the CIO of Orange County, Fla., Mena has had more experience responding to hurricanes than many. In the decade since 2004, a year when Charley and two other hurricanes made landfall in the Orlando area, Mena has led several technology projects to prepare the county for the next megastorm.
In 2011, Orange County introduced an app called OCFL Alert. Developed in-house by the county, the Web and phone app gives people all the vital real-time data they need before and after a hurricane: where sandbags are being delivered, a map of shelters, the location of drinking water, and on and on. Thousands have come to depend on the app for the latest information.
“What makes our applications unique is they are tied to a back end — to case management systems. When you submit something to us, in real time, that information creates a work order or a ticket incident. It’s delivered to the proper department so they can start working on the issue,” Mena said. The user gets real-time updates. “It’s very interactive and it gives the citizen information and it keeps us accountable; it’s transparent.”
An encore is coming. Mena and his staff are working on another app that will help deploy volunteers to big groups like the American Red Cross or the small food bank around the corner. Volunteerism can become chaotic if it isn’t managed well.
When two perpetrators set off improvised bombs on Patriots’ Day the afternoon of April 15, 2013, near the finish line of the Boston Marathon, Justin Holmes and his team immediately jumped into action to alert citizens and keep them apprised of the latest developments.
It wasn’t a matter of creating a bunch of new social media channels and websites. For the most part, the city simply kept doing what it had been doing for years, only more of it. For the past four years Holmes has overseen Boston’s 24-hour phone hotline along with a number of mobile and website applications.
“The wrong time to stand up a social media presence is in the middle of a disaster,” Holmes said. It’s most effective when you have an audience that is already familiar with your brand and leveraging that during an emergency. “Having sustained, proactive and daily engagement with residents helps you in a time of emergency.”
Three were killed and 264 hurt in the marathon bombing. Residents were on edge and looking for any information on friends and family who might be potential victims. People also were sending in tips as law enforcement hunted the two suspects. Thousands of calls simultaneously flowed into Boston’s call center, and the volume remained unusually high for the following two weeks. The city leaned heavily on its scalable telephony to handle the surge of volume.
Superstorm Sandy was a once-in-a-century weather cataclysm of unimaginable proportions when it hit the Eastern Seaboard on Oct. 29, 2012. It hit New Jersey especially hard; coastal developments were wiped away en masse and more than 100 people were dead or missing. Sandy caused an estimated $65 billion in damage nationwide.
For days in advance, New Jersey CIO Steve Emanuel pre-planned and made lists of possible needs — cell sites, email and other emergency messaging platforms, air cards and smartphones were just a few. “One of the things we talked about was voting by mail because it was right around election time,” Emanuel said.
There also were more mundane considerations, like coordinating how relief trucks from across the U.S. would get through the interstate electronic toll system when driving to New Jersey.
Even so, when Sandy came to pass, Emanuel was caught somewhat off guard when his CIO colleagues, like David Behen from Michigan, Karen Robinson of Texas and Hawaii’s Sonny Bhagowalia called him and offered assistance.
“You probably could help,” Emanuel said at the time, “but I need to think about what the hell you can do for me.”
The experience had Emanuel brainstorming that perhaps it’s time to stand up a national database or website where CIOs in need could access a list of contacts, capabilities and resources. It would’ve been useful to know which states could’ve sent New Jersey an emergency supply of spare parts, switches and server infrastructure, or a dozen satellite phones.
Even having another state’s CIO call the telecom companies on his behalf to collect statistics on outages would have been helpful, Emanuel said.
“Something as simple as that, because at one point in time we thought we had 40 to 45 percent of wireless capacity down. We didn’t talk about where or who they were; we put a global number together so the governor would know why he’s not getting information.”
It’s easy to forget that Hurricane Katrina was a truly national-scale disaster. The Category 5 hurricane ravaged New Orleans, but the long-lasting effects extended well beyond the French Quarter into neighboring states.
San Antonio became a destination of last resort for an estimated 30,000 Katrina refugees, and the city quickly realized that an influx would be coming in search of shelter. San Antonio CIO Hugh Miller and a few of his staff members were put in charge of technology at an old Air Force building that would become a processing center for thousands left homeless from Katrina. Some came only wearing the clothes on their back.
It was a hasty operation by necessity, with routers, switches and wires strewn about between cubicles and rows of cots. The first phones were individual outbound wires before Miller’s staff later installed an IP PBX system to organize call routing.
The United Way, FEMA and other organizations got involved, and for a time everyone was doing their own thing. Technicians from the big telecom companies would show up and begin installing equipment without oversight. There were almost too many volunteers. Chaos ensued.
“Early on we realized someone needed to be in charge of these different types of work so that you didn’t have people stepping on top of each other,” said Miller. The city had to aggressively assert authority over IT matters.
Slowly the project settled in and Miller and city staff turned to the day-to-day work of installing big screen TVs, organizing food and a multitude of other functions. They even created databases of the displaced residents and disseminated the information nationwide because the evacuees’ loved ones didn’t know where they were.
“One of the big things that was prevalent was that there was truly not the level of collaboration that was needed,” Miller said. “Granted this was back in 2005, so there were a lot of things maturing — but there was not a clean, integrated way to process people.”
He advised to know who your vendors are and have those key contacts in advance. San Antonio was fortunate to have that list already in place, Miller said, in part because of the high volume of regular purchases necessary for the nation’s seventh-largest city. Consequently, vendors reached out to offer assistance and the city already had extra computers and phones on hand because it was on a scheduled IT refresh cycle.
“We had a regular batch of systems that were ready, so we pulled from that stock and deployed it instantly,” Miller said.
Photo of Justin Holmes via YouTube; photo of CIO Steve Emanuel by David Kidd; photo of CIO Hugh Miller by Steve Towns.
This article was originally published by Government Technology.