The coastal city successfully confronted several data-related issues three years ago when it made major system upgrades, which came with a pretty big lesson: Cities should do all their data backups on a single platform, said Port Angeles Senior Systems Coordinator Jim Harper.
In confronting its data shortcomings, the city in northern Washington state also addressed a variety of structural, legal, procedural and geographic problems, some of which had existed for years.
Port Angeles had already modernized nearly all of its storage away from tape to a digital drive system by Colorado-based STORServer in 2011, but its backup capabilities had remained out of sync with other pressing needs.
The city is less than 100 miles from Mt. Baker, a volcano that last erupted in the 1840s, but which began to emit steam again in 1975, and it backs up to the Olympic mountain range with other potentially explosive volcanoes.
Off the coast of Washington state, the Cascadia fault line lurks — less well-known than California’s San Andreas Fault Line, but believed capable of decimating the entire Pacific Northwest.
Given their relative isolation and proximity to at least two naturally occurring disaster areas, city officials decided Port Angeles needed three data backups, not two, with the third located more than 100 miles away.
“We have multiple possible catastrophe events. All at once. And we do have limited resources, where basically we have a two-lane road going one way and a two-lane road going the other way,” Harper said, referring to the city’s location on the Olympic Peninsula.
Port Angeles’ police officers and firefighters were then in the midst of a migration to virtualized desktops that would make sign-ins easier and let them take their work anywhere. At the time of the upgrade, the agencies had already completed the transition on 50 to 60 desktops but were only about half done. Better data protection would help them too.
City officials, who were embroiled in a lawsuit against a former contractor in 2014, also needed a better way to preserve information on their servers that could be needed in the case.
With all this in mind, officials at STORServer suggested a combination of new and existing hardware —two of its enterprise backup appliances for primary and secondary backups in the city; and for the third, remote backup, adding memory and flash cache to another system the city already owned.
“It was definitely a refresh of their hardware, and it really put them into the most current hardware we were using at the time. One of the great solutions about their disks is they’re easily expanded,” said Katie Nielsen, STORServer’s senior channel manager.
The changes gave Port Angeles 70 terabytes of space in each of its primary and secondary backups, and 30 terabytes of data at the remote site.
The vendor's Virtual Machine Backup software automated data protection for police and fire desktops, backed up multiple machines concurrently — and let them continue operating during backups. Storage Area Network (SAN) storage — essentially, a network of high-speed storage devices — retained data needed to comply with the legal hold.
Primary backup data was kept on disk for a quick restore and to take advantage of the IBM Spectrum Protect’s deduplication feature, which generated a data reduction rate in 2014 of roughly 68 percent.
Currently, with deduplication, Port Angeles now backs up 74 to 75 terabytes of data — but only has to actually save around 24 terabytes of data.
How does it work?
“Normally, when you do a tape backup, you’re backing up one-to-one. What you backed up today, you’re going to back up the same thing tomorrow. Deduplication, it has a matrix in it. It checks archival flags, what has changed and what hasn’t. You’re only backing up stuff that needs to change,” Harper said.
Not having to pay for backup storage, backup tapes and the like has saved the city roughly $200,000 a year on hardware costs, and around $250,000 a year in material, manpower and data security expenses, he said.
Jerome Wendt, president and lead analyst at DCIG, LLC, a storage analyst and consulting firm, noted that the solutions provide feature-rich packages that can be easier to use than the originals.
“They basically put a wizard on top of Tivoli Storage Manager. In essence that’s what they’re doing, writing wizard-like functionality. They script all of that so you just click-click-click. You can do backup every day; you can do it every 10 minutes if you want to. You don’t have to become an expert in virtualized desktop management,” Wendt told Government Technology.
In the tape days prior to 2011, Port Angeles could only back up six servers a day, and file restores had to wait until the weekend. Now, the city backs up more than 165 devices and does incremental work on about 100 of its 210 backup centers each day; full backups on about 15 a day; and is able to back up everything on the weekend.
Saving data needed for its lawsuit helped the city resolve the case in late 2015.
“If the litigation went on for years, we had the ability to restore those servers as a snapshot in time at a certain date,” Harper said. “That was brought out in the negotiations. They said, ‘Well, we didn’t do that.’ We said, ‘Well, we can show that.’ And they settled it. It made a big point in how that case went.”
The senior systems coordinator said STORServer’s solutions put Port Angeles toward the “bleeding edge” of the pack as one of the first cities that officials are aware of with three data backup sites. Arguably the only difficulties in being first came in learning the intricacies of backup — and ensuring it preserves everything, not just what has changed.
“Sometimes being on the bleeding edge of how you’re doing things is not fun because you have to figure things out and hopefully your mistakes don’t bite you,” Harper said. “But the good thing is, if I can learn from it and teach them, then nobody else has to go through it."
