The Show Must Go On

Business continuity planning is helping agencies sustain mission-critical applications in the face of disaster.

by / April 16, 2002
Y2K is long gone, but the remediation efforts agencies underwent are still paying off today.

Y2K remediation required an immense amount of work directly relating to business continuity planning (BCP) -- efforts an organization develops to maintain mission-critical business processes in the face of emergencies or disasters.

Today, agencies are using BCP to prepare themselves for future emergencies and to help them grasp how deeply their mission-critical applications are intertwined with those of other agencies.

All Ears

"We're really changing the focus of IT to be more business oriented than we ever have in the past because of the changes after Sept. 11 and our having to shift money toward disaster plans," said Gail Roper, CIO of Kansas City, Mo. "We've got to open the doors. There has to be a major educational piece to help your customers understand why you're doing what you're doing; why we need to work toward enterprise solutions; why we need to have security in place."

Roper said her department is performing risk-mitigation studies and sharing the information with other agencies, a departure from past processes.

"Before, we understood what the risks were, but we weren't sharing those studies with other agencies," she said. "Y2K was the beginning of the lid coming off of IT and individuals sitting down at the table discussing enterprise requirements. We, as technologists, had a propensity to solve the problem before we knew what the problem was. We've had to develop very strong listening skills."

Y2K forced IT staff to look at problems in new ways, Roper said, including business processes and workflow issues, and that experience is lending itself nicely to business continuity planning.

"The skill requirements have changed," she said. "It's easier to meet the needs of your customers when you understand what their objectives are; what they're in business to do; what their work processes are?"

All Together Now

BCP is one piece of an overall crisis-management plan, said David Morrow, managing principal of EDS' security and privacy services, and that larger plan also includes disaster-recovery planning and human-resource recovery planning.

Though all pieces are important, he said, the BCP element forces enterprises to focus on things that must be done, regardless of whatever emergency or crisis is swirling around the enterprise.

"We did a lot of good work in the Y2K era in getting ready to identify what we need to do," he said. "A lot of organizations, commercial businesses and governments, in the process of that, discovered interdependencies between systems and processes that they never knew existed before."

It's not enough that a state or local government simply identifies a mission-critical process, say welfare payments or related benefits, that must be addressed while assembling a business continuity plan, he said, because those processes don't exist in a vacuum.

"If there's some sort of outside organization or group that has input into that process, or if that process relies upon the input of somebody else, some other state agency, then an agency needs to not only concern itself with the welfare business process and how well that works, but also that other process to make [sure] it continues to work, too," Morrow said.

When commercial organizations began work with EDS, those organizations soon realized the degree of their interconnectedness with others could do as much to cripple mission-critical applications as an in-house BCP failure.

"It's very easy to focus just on your rice bowl, just on the things you worry about," Morrow said. "Especially in the IT environment, programs, applications and processes are so complicated and intricate and involved, it's easy to overlook somebody else having an input to [your mission-critical application] that you didn't know existed."

Details, Details

Once an agency has developed a business continuity plan, though, it's tempting to stick it on a shelf and forget about it, and that could lead to serious problems.

"That's pretty much a sure sign for disaster because just because it's written down in a plan doesn't mean it will work that way," Morrow said, citing an example from his past involving back-up tapes. "Quite often, you'd talk to the guys and say, 'Let's go get the back ups and put them on the system.' You'd get the tapes, put them up or they'd be blank. Or they wouldn't work. Or the tapes were made on hardware that didn't exist anymore, and the old tapes were no longer usable. The devil is in the details in business continuity planning."

Technical details are crucial, and agencies should also pay attention to less obvious details of their defense against a crisis or emergency, said Randy Witt, CIO of Miami-Dade County.

"You have to look at the second and third layer deep in your infrastructure to see problems," he said. "I've had cases in which major communications circuits run diversely out of the enterprise and found out that [the local telephone company] has run them both back through [one] central office. If they lose ... a central office or have a major disruption, you're out of luck."

Witt said his department does its bit to ensure the county's various networks are redundant and the data center is protected, but he lets departments devise their own business continuity plans.

"I try to sponsor activities which will complement departments and help them with reliability and maintainable systems, but if a building burns down, the computers in it aren't going to be worth much," he said. "Whether the department has an alternate facility to go to, that's their business. If they do have one, we'll try to help with making sure the infrastructure is in position there."

Witt said Y2K played an important role in BCP in Miami-Dade County as well.

"We got a lot of value ... out of doing planning for Y2K," he said.

One thing the county is looking at from a centralized point of view is its 911 call center, Witt added.

"We have an alternate 911 center," he said. "We've assessed that really that's not a good enough plan, so we've included a redundant 911 center as an option in an RFP we have out on the street now for a computer-aided dispatch system."

The Right Role

Perhaps the greatest threat to BCP surrounds which business unit of an organization is charged with the responsibility of drafting a business continuity plan, EDS' Morrow said.

"One of the things that I see in IT security and business continuity is that both of them are dependent on the amount of attention you give to them," he said. "We see a lot of problems when we run across an organization that buries the function that worries about these two things."

The problem is that a person or department that has important things to say to the management of an organization about BCP doesn't have sufficient visibility to issue warnings about what needs to be done.

"Security and BCP is a management problem, not a technical problem," he said. "Managers need to be concerned about it as a management problem and have a lot of visibility on that. Quite often, they need to look at their organization to make sure that the department that does this type of planning isn't so low down in the organization or buried so deeply that managers don't hear about a problem until it's too late."