May 12, 2004 By Tod Newcombe
In July 2003, he was appointed CIO of Oregon by Gov. Ted Kulongoski, and was promptly told to get cracking on shrinking the number of state data centers -- and to do it fast. The governor, facing taxpayer revolt, has had to cut, squeeze and thin the state's budget, and high on the list of possible cost savings are the state's 32 data centers. If the state can reduce the number down to a manageable two, according to the governor, it could save between $20 million and $40 million every two years.
If accomplished, the cost reduction will go a long way toward restoring faith in the state government's ability to reduce spending, according to The Oregonian, the state's largest daily newspaper. But cost savings are just part of the story. Data center consolidation can also add value when done properly. It could spur standardization, deliver services more consistently, open access to technology, improve personnel management and leverage technology investments, according to New York state's Office for Technology.
But there are a number of hurdles -- any one of which could derail the best-laid plans -- ranging from lack of funding, to ineffective governance structures and concerned unions. The key to success, experts say, can be found only at the top. "CIO leadership is crucial," said Rick Webb, national director of Accenture's CIO Agenda and former North Carolina CIO. "He or she must bring executive and legislative leadership together along with the agency stakeholders."
Rebuilding the Data Center
Leadership is important for a number of reasons, according to Webb. CIOs must decide what to consolidate, how to consolidate and then must successfully manage the people and processes involved with the consolidation. "CIOs need special communications skills to do that," Webb explained. "Remember, edicts don't always work. It's more important that CIOs use a collaborative approach."
While consolidation can be programmatic or service-oriented -- involving financial systems or e-government, for example -- many CIOs are focusing on data centers and the vast proliferation of servers that has spread in the public sector over the last decade.
"Historically new applications have been the key to the growth in the hardware environment within government," said Craig Harper, vice president of federal operations at BMC Software. "There's a lot of hardware that's been installed to run the many applications that have grown up in government, and much of it is underutilized and expensive to maintain." BMC produces software tools that allow CIOs to model their enterprise and optimize their methodologies for using server farms and data centers. The goal is to maximize performance and help consolidate where it makes sense.
Deciding what to consolidate is a matter of alignment, according to California CIO J. Clark Kelso. "We're simply trying to do a better job of aligning our IT infrastructure so we can manage it more effectively." That means making the centers more customer-oriented in terms of serving the state's departments and agencies.
Though it sounds simple, it is a tough task faced by many CIOs. California is looking at consolidating its two largest data centers, which could save the state $4 million to $6 million annually, according to the state Legislative Analyst's Office.
Webb advocates taking incremental steps, with quick victories in mind, to persuade skeptics that consolidation will work. "The big bang approach to consolidation doesn't always work," he said. "You need both a sprint and a journey to make it happen."
With only two data centers under consideration, California doesn't have much of a choice in how it consolidates.
In Oregon, Fleming is focusing first on the state's 12 largest data centers, which handle 80 percent of the state's data. The rest will come later, once the first phase is completed and everything is stabilized. "The reason for halting the process after consolidating 12 centers is that the more load you put in one center, the more risk there is to the state in terms of recovery," he said.
The impact of Oregon's proposed consolidation on efficiency and productivity is best illustrated in a study conducted by Gartner, the consulting firm. If the state combined its three largest data centers, the combined capacity would be 60 percent of one mid-sized data center built today. In other words, explained Fleming, the state's existing data centers are badly fragmented, inefficient and extremely costly to operate.
More Machines, Fewer Workers?
If leadership is the key to getting data center consolidation off the ground, then good management is the glue that holds it all together. CIOs and their deputies must manage people and processes effectively. More than one private-sector firm has seen its consolidation effort crash and burn because of management problems in these two areas. So imagine what it can be like in the public sector, where independent funding makes it easy for some agencies to ignore the plea to centralize and consolidate their servers, and unions often question the need to downsize the number of managers required to run a consolidated center, despite the fact that this can be where the biggest savings are achieved.
As data centers grew over the years, system administrators were added by the dozen. The traditional rule of thumb, according to industry experts, has been one administrator for every 20 servers. For governments with thousands of servers, the number of administrators quickly adds up. But in recent years, technology has changed that equation. Companies, like Opsware, make software that automates much of the maintenance and administrative work needed to run a data center. Today, the ratio of worker to machines has been reduced to one administrator for every 50 machines -- or 150 in some cases.
But public-sector CIOs say the last thing they want to do is lay off highly-skilled IT workers, especially during the economy's current jobless recovery. Fleming is particularly aware of that issue in Oregon, a state that has struggled during the recession. He emphasized that Oregon will populate the new data center with the "best people we have to get the right knowledge transferred there," adding that the state will make every effort possible to reposition surplus workers, budget funds for retraining, and as a last resort, outsource workers rather than lay them off. Data center consolidation savings will come from eliminating redundant capacity, and adopting new tools to make the center more efficient, he emphasized.
Without addressing the worker reduction issue directly, California's Kelso has one eye on the "tens of millions of dollars coming our way if we do consolidation right," and the other eye on the fact that the two data centers under scrutiny run on dissimilar platforms. That may require keeping more staff on board to run the centers. In fact, Kelso believes California's government is too big to push for massive standardization on a single platform, which would ideally generate the biggest savings.
"I'm convinced there's no reason for us to have one e-mail platform, for example," said Kelso. "Not for the public sector. Not for California. I do think we can reduce the number of platforms we currently use. But one size does not fit all."
Bring on the Stakeholders
While new data centers can be incredibly productive, they are of little value if agencies and departments aren't willing to buy into the concept. Data centers of the past -- in both the public and private sector -- haven't always been reliable in delivering the performance and resources required by agencies and their application developers, according to Johna Till Johnson, president and chief research officer for Nemertes Research. "The net result is that agencies want their applications -- and servers -- in house," she said.
In fairness to data center staff members, their jobs have been made difficult because of the complexity and unreliability of the client-server model government adopted with great gusto, along with corporate America. Till Johnson also said data centers have been constrained by telecommunications contracts locking them into bandwidth requirements that don't meet current or future needs.
The solution is to come up with new governance models that take these factors into account, include the stakeholders, and specify who controls the applications and infrastructure. Oregon is exploring a new governance structure that takes into consideration the state's blending together of data centers from several entirely different agencies, and has to figure how to merge the different talents that exist within each one.
According to Kelso, the key is structuring IT governance so business units still control the applications. To reach that point, CIOs must collaborate with the agencies. "I don't issue edicts," said Kelso. "I'm constantly meeting with departmental information officers. As CIO I don't fall into the trap of becoming the dictator of technology. That's a bad place to go."
California is evaluating numerous governance models. One approach calls for a governing board populated with a majority of departmental managers, who can ensure the data center director stays customer-focused. The second approach, which California may use in combination with the first, is to generate incentives that keep the center customer-focused. "Overall, you need enough structure and incentives in place so the governance mimics the kind of pressure a private-sector data center feels," explained Kelso. Without moving toward a quasi-private-sector model that differs from the current public-sector model of running data centers, consolidation doesn't make sense, he added.
Beyond the big issues of leadership, people and processes, CIOs have a number of factors to weigh as they delve into consolidation. For instance, when consolidation occurs, costs in certain areas can rise instead of fall, according to Till Johnson. As a government reduces the number of centers it runs, network capacity sometimes rises because the remaining data centers are farther apart than before and require more bandwidth to run applications at peak performance, she explained. Applications running on servers farther from users can be sensitive to latency and time issues that affect worker performance, she said. "Getting rid of any kind of latency can be costly."
Some governments are getting around this problem through collaboration, according to Accenture's Webb. "Telecom contracts are expiring in certain areas, so states are collaborating with local governments, colleges, even public school districts to share the next generation of network services that can handle the capacity needed to run consolidated data centers," he said.
Another problem is increased vulnerability. If the number of data centers is reduced, CIOs must be sure they can handle backup and disaster recovery should one of the remaining centers go down. That wasn't such a big issue in the past, when public-sector organizations had many data centers. Now it is, and ensuring business continuity can be an added cost.
If consolidating servers, CIOs must choose between blade servers, racks of 1U (unit) servers or virtual servers. Each has its own advantages and disadvantages. Many consolidation experts advocate standardization to avoid fragmenting and stovepiping platforms, skills and purchasing resources within a center. But as Kelso explained, that might not be possible for large governments and agencies.
The answer to these questions isn't in choosing the right technology, but finding the right methodology to deliver the most bang for the buck, Webb said. "Effective consolidation is about demonstrating value, cost-effective delivery and enhanced outcomes, which is what the public sector needs to be about these days," he said.
Kelso sees the situation a bit differently, although the goals are the same. "One of the things I'm doing is keeping realistic about what we accomplish through consolidation," he said. "In a sense, we are putting ourselves in a better position to manage our IT infrastructure. It's nothing more than that. The data centers are part of the state's strategic plan for IT, but we're sticking with our basic notion that IT has to be driven by business needs and analysis of business re-engineering."
You may use or reference this story with attribution and a link to