NASCIO Midyear: 5 Disaster Recovery Lessons for CIOs

Two battle-tested technology chiefs share lessons from the front lines of Puerto Rico and the Virgin Islands in the wake of Hurricane Maria.

by / April 24, 2018
Angelo “Tony” Riddick, CIO, Virgin Islands David Kidd/Government Technology
BALTIMORE — A rapt crowd gathered in the main ballroom for the morning session on the second day of the National Association of State Chief Information Officers (NASCIO) Midyear conference to hear from a couple of CIOs whose resilience was tested in ways few others has been. U.S. Virgin Islands CIO Angelo “Tony” Riddick and Puerto Rico CIO Luis Arocho talked about the conditions they faced following Hurricane Maria last September.
 
An army veteran, Riddick first experienced combat as a 50-year-old during a brigade changeover in Iraq in 2011. Describing the harrowing events that resulted in a fatality and several injuries to members of his team, he revealed that what the islands experienced after Hurricane Maria made his nearly 400 days in Iraq “look like a baseball game.”
 
“Our infrastructure was totally devastated,” he said, adding that he’d never before witnessed destruction of that magnitude.
 
In Puerto Rico, Arocho talked about the fact that the event elevated the role of the CIO’s office, which used to be a lean team operating out of the governor’s office. “They now realized the importance of technology and the importance of communications,” he said. For the three days following Maria, given how completely infrastructure had been impacted, there was “no government, no decision-making,” he said. Looting and unsafe streets compounded the island nation’s suffering.
 
Though the rebuilding continues, Riddick and Arocho offered the audience some of their lessons learned to better prepare them for future disasters:

Inject some reality into your disaster recovery plan.

“A plan goes to hell when the first round is fired,” Riddick said, referencing his service in the military, adding that the most effective use of an emergency plan in the middle of a disaster might be as a door stop. The two leaders recommended disaster recovery drills that truly test resourcefulness and resilience. “Pull the plug out of the wall and see how you operate. Send some people home and see how you operate,” Riddick advised. 

Enlist a wide circle of collaborators.

Nothing better underlines the critical nature of communications than a disaster on the scale of Hurricane Maria. In Puerto Rico, many key players in response weren’t available after the event. “Eighty percent of the National Guard was not accounted for after the hurricane,” Arocho said. A communications group was quickly assembled, made up of the Governor’s Office, the Puerto Rico Telecommunications Regulatory Board, FEMA, some federal law enforcement agencies and the CIO. 

Back up your backup. 

Riddick recommends a three-tier backup plan, and he has a good reason for the approach. “Don’t do what we did,” he cautions, explaining that their data backup was located in the same building as their primary system. “Our system failed and we’re still trying to recover our data,” he said. In addition to locating backup in a second location, such as the cloud, he now suggests that a third set of data should be housed with the agency customer. 

Be prepared for problems outside your scope. 

Many of the activities Arocho and Riddick described in the recovery process fell into that area of the job description that leaves room for ambiguity: other duties as required. Trying to restore normalcy on an island — where access to resources is limited (worsened by an airport that lacked communications to direct air traffic) — led to challenges like a run on the banks and gas stations by citizens scrambling for cash and fuel, fearful that both would soon be scarce. “Literally, the banks ran out of money … the banks asked us for help,” Arocho said. 

Line up your private-sector partners.

Many people and groups descend on a disaster site, offering help and support, and it can be tough to distinguish between well-meaning volunteers and opportunists looking to profit from others’ suffering, Arocho warned. One group of allies to keep close ties to, though, are industry partners whose technical aid is critical in restoring power and connectivity post-disaster. 
 
Officials in Puerto Rico partnered with Google to bring connectivity via balloons flying at high altitudes using wind currents and machine learning. Another recommendation was to get telcos to enable open roaming between networks — a considerable technical feat. “Make sure that that gets written in your disaster recovery and business continuity plan,” Arocho said. 
 
Noelle Knell Editor

Government Technology editor Noelle Knell has more than 15 years of writing and editing experience, covering public projects, transportation, business and technology. A California native, she has worked in both state and local government, and is a graduate of the University of California, Davis, with majors in political science and American history. She can be reached via email and on Twitter.