In Case of Emergency ...

"After a bio-terrorism event, you will likely not be able to access health services Web sites for information due to capacity constraints, yet you will be able to check your Lotto numbers instead." -- Anonymous senior IT state official commenting on his state's inability to prioritize IT services effectively

by / October 10, 2006
This seemingly ridiculous scenario is fairly likely during a sudden surge in demand for access to information from a narrow set of government services.

Ironically it may not even be one of the more egregious potential ironies for government IT in supporting crisis situations.

How are government IT organizations prepared to respond to increasing threats of terrorism, a pandemic outbreak, or the next natural disaster, such as an earthquake or flood? I am not talking about the National Guard, the Office of Emergency Services, the Office of Homeland Security or the State Highway Patrol -- whose roles are well defined -- but rather the departments of Financial Institutions, Motor Vehicles, Finance, Consumer Affairs, Employment Development and many others that may face the daunting question: How will we continue to operate if a pandemic is announced?

Can government leaders confidently say their CIOs have a plan to keep the work force connected and serving the public?

"Government entities outside the first-responder community have not historically looked at this issue from a risk management perspective," said Kelvin Pye, deputy director of the California Department of Technology Services. "In IT we are typically funded for normal operations, not the extraordinary circumstances of a major event. People may have false assumptions about what capability really exists to respond."

Few, if any, organizations have the luxury of funding response plans for specific emergency situations. In the aforementioned example, it is unlikely -- and impractical -- that health services Web sites be designed for peak capacity based on the theoretical possibility that the need would arise during a pandemic bird flu outbreak or other crisis.

Additionally many state government departments and agencies can hardly deal with the complexity of managing the technology they have today, let alone take on new challenges, such as those presented by homeland security threats or other emergency situations. Estimates show that a typical IT organization can spend 80 percent of its budget simply managing existing systems and applications.

When reviewing mandates across the country, it is generally required by administrative oversight that each department or agency have some form of disaster recovery or operational resumption plans. But these are almost universally limited to isolated circumstances (e.g., a building evacuation). Few consider regional events that may displace not only their staff, but also the staff of allied agencies and departments on which they may be interdependent. Even fewer really test or exercise these interdependencies.

Business continuity or recovery services should be planned commensurately with the potential impact, which is often not fully considered. When the Bank of New York, reportedly the largest government securities broker at the time, experienced a 27-hour IT outage in 1985, the bank was forced to borrow $22 billion, which ended up costing its insurers almost $5 million in increased interest charges alone. That was a direct impact.

Consider the indirect impact of a failure in a state-run eligibility system, like Temporary Assistance for Needy Families. Not only could billions of dollars not make it to families in need, but the impact to the economies in which they live could be considerable, and the local and state government themselves could lose millions in sales tax revenues. Operational recovery should be about risk management, but today it appears to be about filing a report and exercising a script on an annual basis.

But the situation is not hopeless. Two promising broad trends are emerging. First, more organizations are finally giving up on the specific response plans around certain hazards -- e.g., mudslide in Denver, levee break in Sacramento, Calif., bomb threat in Austin, H5N1 Pandemic in Tallahassee, Fla. With the notable exception of the Department of Defense, which maintains extensive war plan options for each region and each scenario, most government entities have no capacity to cover so many permutations, and have found investments in all hazards planning to be far more productive. The second trend involves "emergency preparedness friendly" technologies that have matured over the last decade, driving other business initiatives that have everyday value, not just in the event of the unexpected.

"We can't plan for each hazard individually as well as we can plan for them all at once," said Dan V. Johnson, former director of Minnesota State Homeland Security. "For most agencies, generalized response capability has a much higher return on investment than specific solutions." Johnson contends that it's about time state and local jurisdictions realize more fully that they are already spending a lot of money on general purpose IT investments that are simply not aligned around emergency preparedness.

"For example, e-mail systems that are used every day can be a very effective way of reaching out to all state employees after an event," he said. "Unfortunately many states have not viewed e-mail and other collaboration tools as part of the emergency response efforts and tend to be in silos."

All hazards include those that occur naturally, as well as those that occur as a result of human interaction, both intentionally and unintentionally. Examples include:

  • local disasters (building fire, water main damage);
  • regional natural disasters (wildfire, floods, earthquake, tsunami, pandemic); and
  • man-made incidents (hazardous material spills, terrorist threats).

    Government's reputation for trailing the private sector in IT is usually discussed with a disapproving eye toward change. The flip side to that has typically been a critical view of the private sector, chasing one fad after another -- where are dot-coms now?

    The reality is that neither sector is perfect, but that government may have benefited nicely from a number of investments made from failing business concepts such as the application service provider market and storage service providers. Neither took off as independent business models, but both made major advances in technologies that are now useful to all large enterprises, including government.

    Enter the Resilient Government
    Numerous trends in the way government agencies work and interact may be pivotal to CIOs' ability to build an adaptive, resilient government.

  • Telecommuting: Working from home has become an important aspect of the work/life equation for government. In his testimony supporting the pending Telecommute Act of 2002, Rep. Alcee L. Hastings, Fla., cited homeland security, employee morale, traffic congestion, environmental conservation and productivity as justifications for permitting telecommuting, noting that "45,000 federal employees exercised their option to telecommute for 52 days or more in 2001. ... These federal employees were among the 19 million Americans who telecommuted at least once last year."

    This number continues to grow at the federal level and has skyrocketed in the private sector. According to the most recent census data, the rate of job growth in telecommuting is twice that of conventional on-site positions. State and local governments that allow or encourage it have found they are more competitive as employers for the top talent in the industry.

  • Integration of field operations with central command: Most success stories about mobilizing government teams in the field fall into the category of "point solutions" (so called because they solve just one problem, such as social workers' access to a case management database, but not to the suite of capabilities they would have from "the office"). Unfortunately there are also cases where field staff is saddled with "9-pound pencils" that are only used to capture and print forms previously done by hand, without integration to back-end databases and services.

    Too often, fieldwork in law enforcement, regulatory investigations, health care or other mobilized functions is restricted to the office. Public-sector agencies that untether their staff are finding more work can be done with the same work force with higher levels of satisfaction compared to fieldworkers who must return to their office to do paperwork.

  • Asset efficiency: One emerging trend is the concept of office "hotels" in which employees are considered transitory, in and out of communal office space. One example: The U.S. Patent and Trade Office documented millions of dollars in savings by having more than 50 percent of its eligible attorneys "hotel" in various offices by sharing space on an as-needed basis. But this is only new in government. A decade ago, Michael Bell, then director of corporate real estate for Dun & Bradstreet, estimated that companies could achieve 25 percent to 40 percent real-estate savings by leveraging telecommuting efforts, including "hoteling."

  • Enterprise initiatives: Running a service across the enterprise -- such as a common online payroll tool, financial management system, or e-mail and collaboration solution, with self-service functions that let employees interact with the system and obtain speedy access to accurate information -- is an important trend for all large organizations. The key is that these applications must be available to the entire staff, whether they are sitting on the 22nd floor of an office building in a state capital, working along a roadside as part of a maintenance crew, or watching for early forest fire signs in remote counties. In doing so, organizations ultimately create an enterprisewide user identification and authentication system that can be used for provisioning all services, not just an SAP or Oracle client.

    These trends are not new. Indeed, each has been evolving over the last decade or longer. The point is, there are well established trends based on multidimensional factors that are encouraging a more agile working environment. Emergency preparedness doesn't have to stand alone. CIOs can align factors into a strategic plan rather than rely on happenstance to address the needs around emergency preparedness.

    On the technology front, a set of technologies that enable a mobile, agile and resilient government have emerged and matured in a manner that would allow CIOs to address emergency preparedness as an integrated element of IT investments. "CIOs today have more technologies at their disposal than ever before, and unlike a few years ago, these technologies can now be deployed safely across the public and private sectors," said retired Lt. Gen. Harry Raduege, former director of the Defense Information Systems Agency. "Hackers love organizations that believe their information is safe behind the walls."

    The following technologies have been battle tested in e-commerce, corporate enterprises and across government and have sound business cases surrounding their use outside emergency preparedness:

  • Online, server-based and Web computing: For years, applications have been moving to the Web one at a time as part of the refresh cycle. An increasing number of IT leaders realize that as the trend continues, they are likely to be left with only a few straggling applications delivered locally. Technology now allows all applications to be delivered via the Internet effectively and efficiently -- including ones traditionally not even considered, such as office productivity suites. Depending on the business needs, CIOs have the choice of stateful (rich clients) or stateless (HTML-style clients) to meet their needs. Now a complete suite of applications can be delivered to any connected computer in the world, enabling fully mobile secure computing.

  • Mobile computing device: End devices are becoming more powerful and platforms are converging. You can now have a "smart" cell phone running your favorite operating system and business applications. CIOs have more flexibility in choosing platforms with a philosophy of any device, any time, any place and any application.

  • Wireless gateways: Many corporations have increased productivity and reduced IT investments by eliminating the "horizontal cable plant" of CAT5e cables running through the buildings. Employees can now move around a facility in a connected mode, gaining valuable time previously spent reconnecting to physical telecom ports. Additionally the rapid worldwide expansion of free and low-cost wireless Internet access points -- from airports and coffee shops to state parks -- provides an extended network for fully networked organizations. This gives CIOs another channel for information services delivery. Who would have thought that a coffee shop could figure into an IT strategic plan?

  • Virtualization technologies: Virtualization lets users connect to resources in a manner that masks the physical context of the resources. The more dynamic and automated this connection is, the more "virtualized" the resources and operating environment will seem. Historically this concept was applied to mainframe processing, in which many users would share a common pool of processing capability for economic efficiencies. Today network capacity and traffic routing, including telephony, is largely virtualized, and the ability to virtualize server and storage capacity is greater than ever. CIOs can define resources in pools of network, processing and storage that can be shifted across services when needed. So if a pandemic or other crisis arose, appropriate services could be granted additional capacity while others could be de-emphasized. In a state of emergency, is next summer's online reservation for public campgrounds the No. 1 priority? The point is not that it is or isn't, but that today's CIOs and elected officials don't have the choice.

    These technologies are not emerging: they have emerged. Yet their use in the public sector still trails the private sector because of the many challenges that face public-sector CIOs wanting to build and sustain a resilient organization.

  • Budgeting -- Today, most budgeting is around specific programs, not around generalized capabilities. So it is more likely that funds are approved for education, child welfare or transportation than for collaboration tools, wireless technology, identity management systems and the like.

  • Personnel management -- While many organizations have fledgling efforts to support telecommuting, few outside of the federal government have made a significant effort toward tackling performance management (e.g., moving from attendance-based systems to deliverables-based management) and other issues affecting fairness and costs (e.g., who pays for a DSL line at a government employee's home that is used for both work and personal business?) associated with working outside the office.

  • Legacy -- Even the most ambitious public CIO will have to recognize and accommodate the significant investment in what is currently deployed. There is little chance that all decisions of the past can be revisited quickly. After all, many systems running essential operations in the public sector are more than 30 years old. It is often difficult to justify throwing out old systems and processes to introduce new ones, even if the news ones will ultimately be more effective and efficient.

  • Policy control -- Emergency response is not always presumed to be the role of the public-sector CIO, who is often subordinate to the Office of Emergency Preparedness or another body dealing with crisis management and priorities. For example, in California, the directors of the Office of Emergency Services and Homeland Security convene quarterly to discuss emergency preparedness and continuity of government. This group includes more than 20 high-level state officials from the adjutant general of the California National Guard to the state veterinarian, but does not include the state CIO, the state information security officer, or the director of Technology Services.

    What Can Public CIOs Do Today?
    Realize that neither the federal government nor anyone else is going to drop a trunk of money at your door for your emergency preparedness program. Also realize that you may temporarily need to operate as a skunk works, doing the right thing but without any fanfare.

    Never commit to getting to the perfect solution, but to making sure that each step your organization takes is a step forward. Organizations that wait for the big bang usually find themselves in the same place five years later. Baby steps will advance more quickly.

    Take inventory from a preparedness perspective. Find out where you are already successfully using some of the potentially enabling technologies in your organization today. Celebrate those as case examples.

    Figure out which are the oldest, most temperamental applications or business processes you have in the organization. Establish a small pilot to bring those to the Web.

    Look at your IT procurement policies. See if they are preparedness-friendly. For example, consider the decision-making model around laptops versus PCs. Having more laptops in the environment will not cost much more, but could provide choice and flexibility down the road.

    Look at your IT project approval process. Does the process require that an active project contribute to the overall agility of the organization? If not, then amend it. You can only move forward one project at a time. Make sure that each project helps build your agility and resilience.

    Be open-minded and willing to participate in efforts bigger than you. If you are approached by another governmental entity about sharing infrastructure, say yes when appropriate. Obviously you have to respect your organization's mission and your commitment, but think hard about ways it could work. Don't focus on the 10 reasons it may fail.

    Embrace existing programs that could be extended. For example, if your personnel organization has already incorporated a telecommuting policy, start to exercise it with your own staff. Lead by example. Hold more teleconferences and conduct more online meetings so that remote management skills are more commonplace.

    Work to ensure that every person in your organization has periodic access to the computing environment, no matter how seemingly mundane or unrelated their assignment may be. Make sure they are getting e-mail services.

    Help your colleagues on the emergency preparedness side of the organization look good. They will return kudos and support you.

    In my experience, public-sector CIOs are already doing many of these things, but perhaps rethinking and recommitting to them with the goal of positioning the organization to be more resilient in the future, no matter what happens, is good practice for us all.
    Craig Grivette Special to Public CIO