Moving Forward, Backing Up

Wichita implements a data backup and storage strategy to accommodate growth.

by / February 14, 2003
With information systems managing just about everything, it's hard to picture the volume of data a government organization generates. Even harder to picture is how a government would cope if some of that data -- perhaps payroll, health department or facilities maintenance records -- simply disappeared.

That's why backup and storage are serious business; but they can also become burdensome business as governments rely more and more on information systems.

Wichita, Kan., discovered how burdensome in the 1990s. When the city established its IT department in 1988, it did all computing on a single mainframe. It backed up data to a nine-track tape drive and used paper to record the locations of stored files.

Demands on the system grew quickly. "By 1990, we had four nine-track tape drives and three mainframes," recalled Kevin Norman, Wichita's IT operations manager. That year, the city replaced its paper logs with TAPESYS software to automatically track backups and tapes. But as the enterprise grew, and as it started relying more heavily on PC servers, the number of backup devices and the strategies for managing them mushroomed.

By 1995, Wichita was using three storage devices based on three different tape-drive technologies, Norman said. When the city first started using servers, "for a while, we were backing those up on their own backup systems using their native backup software."

Later, the IT department retrofitted the TAPESYS mainframe software with a third-party solution designed to manage backups from the PCs and servers.

Today, along with the mainframes, Wichita's agencies store about 3 terabytes of data using about 70 servers, 1,000 workstations and 90 thin clients. If Wichita had not pursued a new strategy for backup and storage in 1995, the city could not perform nightly backups on its current information systems, Norman said.


Out of Tapes
Before Wichita made the change, an operator started the backup process at 4:30 p.m. daily and supervised until about 11 p.m., Norman said. After that, the backup system ran independently. Sometimes the operator who checked the system early the next morning found that an autostacker -- the mechanism that moves each tape cartridge into place when the previous one is full -- ran out of tapes before the backup completed. He or she filled the stacker with new cartridges and resumed the backup, hoping it finished before most workers arrived.

As the city added more PCs and servers, completing backups overnight with the hybrid mainframe/PC tape management system became more problematic.

"We were starting to get behind," Norman said. "It was pretty obvious we needed an enterprise-wide backup system that could handle all of our systems."

Wichita sought a Windows NT-based solution to back up both its PC-based servers and its VAX mainframes, to interface directly with a new robotic tape library and fast enough to complete all backups overnight. The only product to meet all these criteria came from SSSI, now called STORServer Inc., in Colorado Springs, Colo.

With STORSever's current line of backup appliances, users can purchase an entire solution from one source and turn to one vendor for service and support, said STORServer's President John Pearring, adding that past users bought storage solutions in pieces from different vendors; when something went wrong, each vendor blamed the others' products.

Today, with one service contract on the entire solution, "We actually solve the problem for the customer after we diagnose it," Pearring said.

Wichita bought its system and service contract from value-added reseller Dataedge Solutions. At the time, STORServer did not offer its solution as a preconfigured appliance, Norman said.

"They'd come in, look at what you were doing and configure you a system," he said.

The city's configuration included a backup server running on Pentium II 500 MHz dual processors, a 40-slot robotic tape library using digital-linear-tape (DLT) technology, software for managing both the mainframe and server backups, and software from STORServer for coordinating the entire process.


Full and Incremental
The server backup software -- IBM's ADSTAR Distributed Storage Manager, now called Tivoli Storage Manager (TSM) -- brought major change to the city's backup procedure.

In the past, Wichita -- like many enterprises -- used a combination of full and incremental backups. Once a week, it backed up all data on its systems, taking a snapshot of the current information. On subsequent days, it backed up only the files that changed in the past 24 hours. On day seven it did another full backup. Without periodic full backups, Wichita had large volumes of tapes holding many generations of changed files; to restore a file, operators had to hunt through those tapes to find the right one.

TSM requires just one full backup, after which it makes incremental backups and automatically tracks storage locations. When someone needs to restore a file, TSM locates it in a pool of storage media. Using STORServer's software, a manager sets policies to indicate how long the system should store different kinds of files. After the expiration date, the system automatically writes over that file; this limits the number of tapes needed.

STORServer's system first backs up data to an array of hard drives, then transfers it to the tape library. An operator who needs to restore a file backed up the previous night can retrieve it faster from the hard drives than from tape, Norman said.

The software also changed the way Wichita provides for disaster recovery. In the past, Wichita stored its backups in an off-site vault; if a file needed restoration, someone retrieved the appropriate tape from that location. But TSM automatically makes two copies of each backup. The city keeps one set of tapes on site where they're handy for restoring on short notice while the other set is in the vault, safeguarding the data in case of a disaster.


A Half-Hour Job
The new system requires less time for daily backups, uses fewer tapes and requires much less operator participation. Instead of supervising the backup until 11:00 each night, an operator spends 30 minutes in the evening removing blank tapes and adding new ones as needed. During the day, an analyst checks the system to make sure it's performing correctly.

"My evening operator job is gone," Norman said, noting that loss saves the city the cost of an additional worker. "That position has been deployed to do other things."

Although the new backup system works faster, the process still runs all night. That's mainly because the city has added so many information systems and servers since 1996.

"We're still struggling to meet our backup window, and we always will," Norman said. "If we had it to the point where we could actually back up everything in two or three hours, we've overbought."

To meet the challenges of its ever-expanding information-services enterprise, Wichita has made two upgrades to the system since 1996. In December 1999, it replaced the tape library with an 80-slot system using AIT-2 technology, and early this year it replaced the entire backup system with a more powerful one, including a Compaq ProLiant ML530 server with 24 72.8 gigabyte SCSI hard drives and a 180-slot AIT-3 tape library.

Rather than buying one of STORServer's backup appliances, Wichita worked with Dataedge to specify the exact system it wanted, using faster equipment than STORServer offered at the time, Norman said.

Norman could not locate records indicating the total cost for all components of the original system. But he said the hardware cost $25,000, and the city has spent about $30,000 per year for maintenance. For the new system, the city is paying $42,000 for the server and all its disk drives and $90,000 for the tape library, including the drives and tapes. Those figures include a total of $15,000 for consulting services, installation and data migration.

Wichita has not tracked the return on its investment in the new backup technology. Payback wasn't even part of the justification, Norman said, explaining that the main goal was to build a backup system that could grow with the organization.

"We try to anticipate obsolescence and growth in demand for new systems," he said. "Otherwise, systems are replaced on an emergency basis, which can shorten the lifetime of the solution."

Bio: Contributing Writer Merrill Douglas is a freelance writer based in upstate New York. She specializes in applications of information technology. E-mail: mdouglas@stny.rr.com
Merrill Douglas Contributing Writer