Counties have many reasons to consolidate -- enhanced efficiency, reduced operational costs and easier-to-manage infrastructures -- especially when they're as large as Fulton County, Ga., which includes the city of Atlanta. According to the most recent U.S. Census Bureau data, the county is home to nearly 1 million residents. The county's Department of Information Technology (DOIT) undertook an enterprise server consolidation project to make serving such a large constituency -- and the county's 42 government departments the DOIT supports -- a little less trying.
"One of our biggest concerns was obviously the server sprawl that was happening," said Ryan Fernandes, the CIO of DOIT. Before the consolidation project began in 2005, many county applications were installed in different locations. Prior to that, many departments had applications running on their own servers instead of one central machine. "We decided that it made logical sense for, No. 1, all of those servers to come under the IT [department's control], and then at another point in time, they were physically moved to our data center. Most of the servers were physically moved to the data center [after the consolidation got under way]." Once the migration happened, the county could more easily audit their server environment and assess total cost of ownership.
At first, Fulton County officials didn't believe a server consolidation project was necessary. In 2005, IT decision-makers decided to replace legacy server equipment to increase efficiency and capacity. In the process, they discovered consolidation would be a valuable improvement.
"The original plan, as my predecessor saw it, was to just simply buy blade servers as a replacement for hardware that had reached [its] end of life," said Jay Terrell, the county's chief technology officer. "It was not a consolidation strategy; it was just a shift of platforms."
But after the decision was made to replace equipment, the county quickly saw the benefits of virtualization and consolidation. When virtualization software is installed on a server, the technology allows one server to house multiple "virtual servers." Thanks to this technique, Fulton County's blades can house more servers on one machine.
"For every blade, we could put eight servers on it instead of 1-to-1 servers to the blades, so at that junction, we moved away from a hardware-centric approach to a more virtualized-centric approach," said Keith Dickie, assistant director of networks for the county.
Before the hardware replacement and consolidation decisions were made, the DOIT managed various department servers -- Terrell estimated about 250 -- in remote locations across county infrastructure. Not all servers were being used to optimum capacity; and to make matters worse, they weren't all being backed up properly, which put a serious damper on disaster recovery and business continuity capabilities.
"You would have one application running per server because that's the way in which it was purchased and funded," said Dickie of the older server environment. It wasn't uncommon for a department to have its own server housing only department-specific applications, which amounted to extra, unused capacity. "So by the consolidation, we were able to take advantage of the [economies of scale] and the power of the virtual consolidated environment to take away that type of wasted resource."
On the Blade's Edge
Fulton County partnered with Fujitsu to acquire blade server technology and use the power of VMware virtualization software. The DOIT consolidated the old servers onto Fujitsu's Primergy series server blades. The IT department also purchased Fujitsu's Itanium Primequest servers, which use Red Hat Enterprise Linux to support Oracle databases.
"There were a lot of surprises [in testing and training] because we were used to dealing with stand-alone servers that were a pretty much a known quantity to us, and we understood how they
connected to the rest of the infrastructure," Terrell said.
The county discovered that not all systems were candidates for virtualization because some crucial systems aren't well suited for the virtual frontier. Dickie said applications like the Kronos time-keeping program are too specialized and require connections or hardware that won't work in virtual environments.
Also, the county's interactive voice-response (IVR) systems can't be virtualized because they need unique hardware to interface with phone systems. These IVR systems need access to the primary-rate interface, which is a digital network that sends voice and other data over digital or nondigital telephone lines.
Manuel Martull, Fujitsu's senior director of server marketing, said Fulton County's upgrade and consolidation allowed IT personnel to maximize its server usage.
"They noticed that some of these servers were significantly underutilized, so the utilization rates were in the 30 to 40 percent [range], and by virtualizing, some of these environments now can run the multiple applications within a single server and boost that utilization rate up to about 80 percent," he said.
Martull said the Primergy equipment will provide the county with dependable operability with very few failures over time.
"Once you purchase a Primergy system, it is expected to perform without failures for up to six years," Martull said, which would be a significant improvement compared to the server equipment Fulton had before the consolidation. "Some of the older systems that they had, I think they were 7-year-old systems."
Upside of Consolidation
The county installed four Fujitsu BX600 S2 chassis each containing 10 BX620 S2 blades. Fulton County began the server blade implementation in January 2006, implementing the first two blade chassis and installing VMware software. The county purchased two more chassis later that year. After purchasing the virtualization solutions, county IT personnel began testing how to use and configure the technology for different scenarios.
"For our testing environment, we used one blade from each blade chassis, and each blade chassis has 10 blades stored in them," said Katrina Terrell, a network manager for the county who is not related to her colleague Jay Terrell. She and her co-workers were able to test the scenarios differently and in different environments.
Using only one blade from each chassis lets DOIT take only a portion of the server environment for testing, while leaving the rest available for application support. This agility in the server infrastructure allows the DOIT more flexibility for day-to-day operations.
The county took a slow and steady road to implementing the blades and the virtualization, not to mention testing, configuration and training. The project was completed in fall 2008. Jay Terrell and his co-workers are unsure exactly how much the consolidation cost the county because of the implementation's long time frame.
The new setup improved the county's ability to manage its server environment in a few crucial ways. For one thing, the blade technology saves the county maintenance hours because it allows remote upkeep. And because the technology is standardized under the DOIT and not dispersed among multiple departments, it's easier to maintain. Virtualization also lets the county clone new servers from others much faster than before.
"When it comes to new server environments, we could have a server up within an hour or so or less," Katrina said. "It definitely enhances our environment to be able to reach the level of service to our departmental customers and maintain that level of service in a timely manner."