All real techies know that badness is going to happen -- maybe not today or tomorrow, but it's definitely coming down the pike.
Evansville, Ind., Mayor Jonathan Weinzapfel and Vanderburgh County, Ind., county commissioners decided to plan for disaster, rather than react to a future occurrence. In August 2005, Evansville and Vanderburgh County consolidated existing computer network components into a clustered solution.
In a traditional network, when a server fails, disaster strikes. At least a portion of the network will assuredly go down. Both money and customer faith are lost -- data is corrupted and staff and clients are inconvenienced -- while IT staff frantically troubleshoot the cause of the failure.
For Evansville, one of the catalysts for change was one basic business decision: What does it cost for the city to be without a critical service? If the enterprise cannot go without a service for an entire business day, then that service is a candidate for clustering.
"Citizens have come to rely on e-mail to communicate with the elected officials and departments," Mark Rolley, CTO of Evansville and Vanderburgh County, cited as one example.
Stable e-mail, 24/7, is critical, and is therefore considered a viable candidate for clustering within Evansville and Vanderburgh. The issue was addressed head-on by the jurisdictions as a result of an efficiency study conducted by city and county executives.
"After a lengthy review of our current operations and support contractors, we started a consolidation effort in late July 2005, and it still continues today as efficiencies are identified through internal review and auditing," said Matt Arvay, CIO of the Department of Information Technology, which serves the city of Evansville and Vanderburgh County.
The consolidation efforts included rearranging old, duplicate equipment to create a clustered solution that would enhance the availability of all critical services.
Defining a Clustered Solution
A clustered solution includes a system designed to have all critical components redundant, and the redundancy allows a single component to fail without taking the entire system offline, Rolley explained.
"If a hard drive were to physically fail, another standby unit would automatically take over," Rolley said, noting that a clustered solution minimizes downtime because the solution is built around an anticipated and planned failure.
In other words, a secondary server is now in place and ready to pick up the ball if the primary fails. If a department were transferring a file at the time the primary server crashed, that file would simply bypass the first server and continue to transfer to the secondary server. This process would have absolutely no effect whatsoever to the end-user, and buys IT staff critical time to diagnose and fix the issue with the primary server.
But clustering doesn't eliminate the need for regular data backups, according to Rolley. "Clustering does protect the data much more from a survivability standpoint, but does not address historical data needs," he cautioned.
If Susie in Community Development accidentally deleted a critical file that last existed two weeks ago, that file would still need to be retrieved from an archived backup. Redundancy is a replication of real-time, dynamic data, and is not a good substitute for a historic archive.
The city/county clustered system concentrates on various levels of fault tolerance, but every operation is performed on two or more duplicate systems, so if one fails the other can take over.
The Evansville/Vanderburgh County cluster uses 40 servers running Windows NT and 2000. The LAN provides an IP network that gives 100 Mbps full duplex to the desktop that connects to a gigabit backbone. WAN solutions vary from dial-up for remote locations that are unable to get faster service to 50 Mbps optical solutions allowing video arraignment between the jail and courts.
Paying for Clustering
"Most government organizations look at the bottom line and not the long-term benefits of reliable systems," Arvay said. "In addition, the consolidation effort of support and systems ... will allow a great number of government agencies to utilize these systems, which helps justify the ROI."
Evansville and Vanderburgh County paid for the initial cost of consolidating into a clustered solution by re-evaluating the entire network piece by piece. Arvay and Rolley asked engineers to imagine that all existing components lie on the floor, and to create a plan that envisioned an effective, highly available network from those parts -- focusing on availability and survivability of existing systems and processes.
They realized a cost savings by not upgrading and maintaining separate systems, which then allowed them to upgrade one system to a cluster. Any excess parts were put up for sale via government surplus auction.
In addition, the IT budget was reduced by reviewing several expensive hardware and software maintenance contracts that were no longer necessary as a direct result of the consolidation/clustering effort. In some cases, the level of maintenance purchased annually became unnecessary because those devices were covered by standbys under the new clustering system.
Finally the mayor and County Commissioners Cheryl Musgrave, Bill Nix and Tom Shetler Jr., and former Commissioner Rep. Suzanne Crouch allowed Arvay to reinvest year one savings from the consolidation efforts back into the system to improve it.
Both Arvay and Rolley agree that the consolidation and switch to a clustered solution would not have been possible without strong support and leadership from executives that placed a priority on identifying opportunities to increase efficiencies within government, including efficiencies in IT.