It's only important when things go haywire -- whether another hurricane season is looming off the Southern coastal states, nasty tornadoes are wreaking havoc in the Midwest, or there's river-swelling rainfall in the West.
A solid disaster-recovery plan is designed to breathe life back into a devastated city. Even though such events can disrupt many cities and counties at once, it's typically every jurisdiction for itself when it comes to getting back on track.
That mindset is changing, especially in planning for disaster recovery in IT. Governments have started working together to make sure each other's mission-critical applications are available after a disaster hits.
No Gambling in Nevada
In early April 2006, Nevada officials announced the Nevada Shared Information Technologies Services project (NSITS), designed to radically reshape the way jurisdictions protect critical applications from failure in case of a disaster.
Terry Savage, Nevada's CIO, said four agencies are currently involved in the project -- the state's Department of Information Technology; Clark County, Nev.; Las Vegas; and the Las Vegas Metropolitan Police Department. Other public-sector entities in the state will be invited to participate in the NSITS once the project is up and running.
Savage said Las Vegas, on behalf of the three other partners, released an RFP for a feasibility study examining the alternatives for deploying the NSITS. The study's results are expected by September 2006.
Savage said the Las Vegas area seems ideal for a shared-use facility for state agencies because of its distance, nearly 450 miles, from the state capital of Carson City -- given one of the first rules of disaster recovery is that the chosen facility must be far enough away from the disaster zone so as to not be damaged by the same event.
Conversely Carson City is the ideal place to locate a facility to safeguard Las Vegas' data and applications for the same reason.
For now, the details of what form the NSITS will take and where the shared-use facilities will be housed are necessarily hazy, Savage said, because the four agencies won't act until the study is completed.
"Conceivably the feasibility study could come back and say, 'You guys are nuts. This isn't feasible. You can't do it,'" Savage said. "We don't believe that, obviously, but if the study does come back that way, we'll go do something else."
The partners will wait until the study is complete before defining funding options and guidelines for customer use, or making location and building plans.
"If the study comes back with the finding that this is something that can work, then we'll start looking at actual requirements and definitions, and then play out the various location options against the requirements and what meets the requirements best for the minimum cost," he said. "We won't get to that stage before early 2007."
Savage said the 2007 legislative session is the target for seeking funding for the shared-use facilities envisioned by the NSITS project.
Form and Function
The NSITS builds on earlier work done by the state in creating a communications steering committee to push interoperability in Nevada radio communications.
"We've been working with a number of jurisdictions around the state in radio communications interoperability for several years," Savage said. "Last year, we got the first communications interoperability plan ever in the state passed by the Nevada Homeland Security Commission.
"It was the same kind of thing," he said of the Nevada Communications Steering Committee. "It was people from counties, cities, north, south, rural, urban -- a whole spectrum of stakeholders that came together and came up with a plan that was passed out of the committee without dissent, and passed by the commission unanimously and without amendment. That took us about three years to do."
Savage said the hope is to draw on that earlier success and attract an even broader group of participants in the NSITS project, and that those who become members will get a better, more reliable and less costly disaster-recovery system.
Aside from the practical considerations of building the shared-use facilities, NSITS members will have to iron out operational considerations.
"One of the big issues in any project like this is going to be governance," he explained. "How do you run the thing? Does it make more sense to farm it out? Does it make more sense for one or the other of the agencies to run it and bill it out to other agencies? We don't have an answer for that yet.
"The study, ideally, will identify the best way to do that, and I'll go along with whatever the numbers say," he continued. "I'm a numbers kind of guy."
Numbers also matter in Clark County, where more than 38 million people visited in 2005 to get a taste of Las Vegas' entertainment.
Rod Massey, the county's CIO, said the county joined the NSITS because of the same bottom-line pressures facing all levels of government -- creating a more efficient operating environment to deliver better services to constituents while saving money.
"We could all go our separate ways, build separate facilities for locating computer equipment and technology or disaster recovery, or we could partner," Massey said. "We all have similar needs, certainly we have our unique elements, but looking at this collaboratively, what we're doing is getting the biggest bang for the citizens and taxpayers."
Like other governments, Clark County is rethinking its approach to disaster recovery, Massey said, because recent disasters, such as Hurricane Katrina and 9/11, serve as stark reminders that the old way of to each his own doesn't work anymore.
"From a disaster-recovery perspective, those things make you think, 'You know, you're not an island unto yourself,'" he explained.
Nevada Gov. Kenny Guinn set the tone when he officially announced the NSITS.
"As we learned from the Katrina disaster, computer systems caught in the middle of such a catastrophe are prone to failure," Guinn said in a statement. "A system that allows government agencies to share critical information will go a long way to assuring that information is not lost."
Every government is worried about protecting its core systems, Massey said, and the NSITS' suggestion of shared-use disaster-recovery facilities makes sense, though nobody is ignoring an important caveat -- the feasibility study will have to clearly define the business case and the terms for making the shared-use approach work for NSITS members.
"The study will define what's in it for everyone," Massey said. "You have to be clear on that, because if you want elected officials and executives to be able to make good decisions, then you need to give them the right information. That starts with understanding what's really the business value here."
He said that means explaining to elected officials and executive officers the consequences of taking a particular approach to disaster recovery -- the financial impact to a community and the risks to the capability and functionality of core government IT systems -- whether it's shared-use approach or to each his own.
In League
In North Carolina, partnership will also play a role in how some jurisdictions approach disaster-recovery planning.
rability plan ever in the state passed by the Nevada Homeland Security Commission.
"It was the same kind of thing," he said of the Nevada Communications Steering Committee. "It was people from counties, cities, north, south, rural, urban -- a whole spectrum of stakeholders that came together and came up with a plan that was passed out of the committee without dissent, and passed by the commission unanimously and without amendment. That took us about three years to do."
Savage said the hope is to draw on that earlier success and attract an even broader group of participants in the NSITS project, and that those who become members will get a better, more reliable and less costly disaster-recovery system.
Aside from the practical considerations of building the shared-use facilities, NSITS members will have to iron out operational considerations.
"One of the big issues in any project like this is going to be governance," he explained. "How do you run the thing? Does it make more sense to farm it out? Does it make more sense for one or the other of the agencies to run it and bill it out to other agencies? We don't have an answer for that yet.
"The study, ideally, will identify the best way to do that, and I'll go along with whatever the numbers say," he continued. "I'm a numbers kind of guy."
Numbers also matter in Clark County, where more than 38 million people visited in 2005 to get a taste of Las Vegas' entertainment.
Rod Massey, the county's CIO, said the county joined the NSITS because of the same bottom-line pressures facing all levels of government -- creating a more efficient operating environment to deliver better services to constituents while saving money.
"We could all go our separate ways, build separate facilities for locating computer equipment and technology or disaster recovery, or we could partner," Massey said. "We all have similar needs, certainly we have our unique elements, but looking at this collaboratively, what we're doing is getting the biggest bang for the citizens and taxpayers."
Like other governments, Clark County is rethinking its approach to disaster recovery, Massey said, because recent disasters, such as Hurricane Katrina and 9/11, serve as stark reminders that the old way of to each his own doesn't work anymore.
"From a disaster-recovery perspective, those things make you think, 'You know, you're not an island unto yourself,'" he explained.
Nevada Gov. Kenny Guinn set the tone when he officially announced the NSITS.
"As we learned from the Katrina disaster, computer systems caught in the middle of such a catastrophe are prone to failure," Guinn said in a statement. "A system that allows government agencies to share critical information will go a long way to assuring that information is not lost."
Every government is worried about protecting its core systems, Massey said, and the NSITS' suggestion of shared-use disaster-recovery facilities makes sense, though nobody is ignoring an important caveat -- the feasibility study will have to clearly define the business case and the terms for making the shared-use approach work for NSITS members.
"The study will define what's in it for everyone," Massey said. "You have to be clear on that, because if you want elected officials and executives to be able to make good decisions, then you need to give them the right information. That starts with understanding what's really the business value here."
He said that means explaining to elected officials and executive officers the consequences of taking a particular approach to disaster recovery -- the financial impact to a community and the risks to the capability and functionality of core government IT systems -- whether it's shared-use approach or to each his own.
In League
In North Carolina, partnership will also play a role in how some jurisdictions approach disaster-recovery planning.
The North Carolina League of Municipalities (NCLM) has been studying the idea of a shared-use facility for disaster recovery for approximately the last year and a half, said Lee Mandell, the NCLM's director of IT and research.
The rough vision is for city and county associations in North Carolina and neighboring states to have access to a shared-use facility that's co-owned by participating organizations.
"We would be putting up an infrastructure that would allow multiple leagues to share the overhead, cost and infrastructure of a centralized disaster-recovery solution, and have it available to us whenever one of us needed it," Mandell said. "We're all in hurricane country down here."
As part of examining the feasibility of a shared-use facility, Mandell said the NCLM gauged the fees other leagues paid to specialized disaster-recovery service vendors, and concluded that by pooling those fees, the leagues could build a facility more specifically designed to meet their needs.
"There are two levels to our long-range plan," he said. "One is the partnership among municipal leagues, which is an important piece of the plan and our first step." The other step is proving out both the technology and the operational concept behind the idea -- to perhaps offer access to the facility as a service to our municipal members, Mandell said.
The service option is only being studied for now, and not necessarily something the NCLM will do, Mandell cautioned, adding that many other aspects of such a facility remain to be tested.
"How scalable is this? We're working on the premise that only one league at a time will need the facility," he said. "Therefore, it's only scaled to that size. But once you add a sufficient number of potential users to it, then you have to scale it to handle two or three -- especially if you're doing cities within a particular state, many of which could be hit by that same hurricane."
Mandell said one of the benefits of having many leagues in different states as co-owners is the odds are low that all of them would need access to the facility at the same time. Currently the Municipal Association of South Carolina is working with the NCLM on a pilot of the shared-use facility.
"Once we get it operational and feel like we've worked out all the bugs, we'll be offering it to additional partner leagues to come in at a later date," he said. "We don't intend it to be a two-league solution, but we figure that's enough to get us to the point where we feel it is a viable concept to expand more broadly."
Going Co-op
Farther south, eight local governments formed a disaster-recovery co-op called the Florida Disaster Recovery Collaborative, said Bob Hanson, CIO of Sarasota County, Fla.
The participants have spent the last 18 months testing the disaster-recovery collaborative concept and sorting out issues associated with sharing infrastructure.
Currently Hanson said the predominant players in the collaborative are the city and county of Sarasota, Martin County, Collier County and Leon County, representing a diverse set of geographies.
"We're making good headway down that path in identifying what the space requirements might be and what equipment requirements might be that would be owned by the consortium itself," Hanson said. "We're moving toward formalizing this thing as a nonprofit entity that would own some assets we can deploy to an area that's hit, as well as share."
Network connectivity between the sites was the chief problem, especially in performing live synchronization of data or daily full backups electronically to another site. Hanson said the collaborative estimates a backbone capacity of several hundred Mbps will be needed to create "warm site" capacity wanted by collaborative members.
w some jurisdictions approach disaster-recovery planning.
The North Carolina League of Municipalities (NCLM) has been studying the idea of a shared-use facility for disaster recovery for approximately the last year and a half, said Lee Mandell, the NCLM's director of IT and research.
The rough vision is for city and county associations in North Carolina and neighboring states to have access to a shared-use facility that's co-owned by participating organizations.
"We would be putting up an infrastructure that would allow multiple leagues to share the overhead, cost and infrastructure of a centralized disaster-recovery solution, and have it available to us whenever one of us needed it," Mandell said. "We're all in hurricane country down here."
As part of examining the feasibility of a shared-use facility, Mandell said the NCLM gauged the fees other leagues paid to specialized disaster-recovery service vendors, and concluded that by pooling those fees, the leagues could build a facility more specifically designed to meet their needs.
"There are two levels to our long-range plan," he said. "One is the partnership among municipal leagues, which is an important piece of the plan and our first step." The other step is proving out both the technology and the operational concept behind the idea -- to perhaps offer access to the facility as a service to our municipal members, Mandell said.
The service option is only being studied for now, and not necessarily something the NCLM will do, Mandell cautioned, adding that many other aspects of such a facility remain to be tested.
"How scalable is this? We're working on the premise that only one league at a time will need the facility," he said. "Therefore, it's only scaled to that size. But once you add a sufficient number of potential users to it, then you have to scale it to handle two or three -- especially if you're doing cities within a particular state, many of which could be hit by that same hurricane."
Mandell said one of the benefits of having many leagues in different states as co-owners is the odds are low that all of them would need access to the facility at the same time. Currently the Municipal Association of South Carolina is working with the NCLM on a pilot of the shared-use facility.
"Once we get it operational and feel like we've worked out all the bugs, we'll be offering it to additional partner leagues to come in at a later date," he said. "We don't intend it to be a two-league solution, but we figure that's enough to get us to the point where we feel it is a viable concept to expand more broadly."
Going Co-op
Farther south, eight local governments formed a disaster-recovery co-op called the Florida Disaster Recovery Collaborative, said Bob Hanson, CIO of Sarasota County, Fla.
The participants have spent the last 18 months testing the disaster-recovery collaborative concept and sorting out issues associated with sharing infrastructure.
Currently Hanson said the predominant players in the collaborative are the city and county of Sarasota, Martin County, Collier County and Leon County, representing a diverse set of geographies.
"We're making good headway down that path in identifying what the space requirements might be and what equipment requirements might be that would be owned by the consortium itself," Hanson said. "We're moving toward formalizing this thing as a nonprofit entity that would own some assets we can deploy to an area that's hit, as well as share."
Network connectivity between the sites was the chief problem, especially in performing live synchronization of data or daily full backups electronically to another site. Hanson said the collaborative estimates a backbone capacity of several hundred Mbps will be needed to create "warm site" capacity wanted by collaborative members.
Hanson recently worked out a deal with FiberNet, a division of Florida Power and Light (FPL), to provide Sarasota County with a Gigabit, fiber-optic connection to the Internet and access to FPL's statewide fiber-optic network.
"That gives the access within a couple of miles of, say, Martin County's data center, as well as closer to some of the other counties," Hanson said. "That couple of miles is easy to bridge so Kevin [Kryzda, CIO of Martin County] and I will be sitting with FPL to discuss a similar deal for him, which will give us Gigabit capacity, point to point, across the state.
"That really helps us ramp up and achieve the overall vision," Hanson continued, noting that the deal with FiberNet and the FPL allowed the collaborative to avoid spending significant money to lay fiber-optic cable to create its own backbone. "Now we can do a statewide, Gigabit disaster-recovery network -- and collaboration network in general -- pretty easily."
Recovering Communication
The Charlotte County Clerk's Office is one of the first disaster-recovery customers of the collaborative, Hanson said. Within several months, the Clerk's Office will be able to tap Sarasota County's data center to recover data.
He targets the first quarter of 2007 as the time when Sarasota, Collier, Leon and Martin counties will have equipment installed in each other's data centers for disaster-recovery capabilities. Hanson said he's pleased with the speed at which the collaborative has formed and with members' willingness to address disaster recovery in a new way.
"The hurricane situations of the last couple of years have really put us all on the edge of our seats," he said, "and we're very anxious to resolve some of these things to all of our advantage.
"People are starting to look at disaster recovery differently because the costs were so exorbitant to set up any reasonable recovery capability," Hanson continued. "If you were trying to do it yourself, you're building a fully redundant facility -- basically doubling your infrastructure investment -- and most of the time, you're doing it within the same community that you have your primary facility, which is not the best situation."
If local governments focus on building only one strong disaster-recovery facility in their community and then share that facility's capacity with other governments, he said, everybody avoids doubling their respective infrastructure investments.
Disaster recovery is more than a local government merely knowing its data is safely stored in a remote site. It's about communication, Hanson said, especially when evacuations before a hurricane turn cities into ghost towns as residents flee inland.
"Those people want communication," he said. "They want to know what's going on. They want contact with their commissioners and so forth. One of the critical systems that emerges, surprisingly to many people, is Web capabilities -- an Internet site that can easily be updated."
If Sarasota County's network is completely wiped out, county residents forced to leave their homes could still access information on the Sarasota County Web site, Hanson said. However the site would be running on Martin County's equipment, operated by Martin County staff.
"What we're really restoring is the ability to communicate via that Web site," he said. "If you look at the Louisiana situation, that was the thing that stressed people the most -- the inability to communicate."
That is what drives communities to purchase satellite phones that will operate in the aftermath of a hurricane. That's why Sarasota is part of the collaborative, he said, to be able to restore what might seem mundane, but is used most regularly -- the Internet.
"No matter where a constituent has evacuated to, they know our Web site address," he said. "They can go to that site, get the latest information, and communicate back to us."
