Storage Demands Increase Need for Better Management

As storage hardware costs drop, information management challenges grow.

by / November 24, 2008

Gone are the days when an agency could store all of its data on a small server. Industry experts report that storage needs are increasing between 30 percent and 50 percent annually. This trend will only soar as the size of multimedia and other file types continues to explode.

Fortunately as the demand for storage space rises, the cost of storage media declines. Samsung, for example, currently offers a 1-terabyte hard drive for less than $200. Western Digital, Seagate and Hitachi offer comparable disks for similar prices. A standard 500 GB hard drive typically costs about $100.

The prevailing thought used to be that there's no such thing as having too much storage. So with the cost of storage dirt-cheap these days, why isn't the public sector jumping at the opportunity to store everything and anything? Jim Burns, CIO of Alabama, said cheap storage comes with its own problems.

"It's a bad idea to offer unlimited storage because management costs skyrocket and backup and retrieval becomes unmanageable," Burns said. "Cost of storage media, such as disk, is a very small part of the overall costs associated with an enterprise storage solution. The industry agrees that disk is cheap, but for every dollar spent on disk, it takes about $5 to $7 to manage it."


Infrastructure Costs

The challenge isn't storing massive amounts of data -- the real problem lies in managing that data. IT experts and analysts point to a number of information-management issues facing government agencies: How is data secured, backed up and retrieved? How often are tape backups corrupted? How much staff time is spent on manual restores? How do you access and verify the data?


Duplicate Data

Duplicate data is a huge issue for the public and private sectors. For example, both human resources and payroll might utilize a particular document at the same time, yet the legal requirements for data retention of that particular document may be vastly different based on the department and the governing entity.

Even worse, individual users may keep multiple copies of a particular file, perhaps in various stages of development. These files are shared with other staff via e-mail, which in turn also automatically saves each version of the file. Before you know it, there can be dozens, if not hundreds, of copies of the exact same file being backed up every night. Multiply this process across hundreds or thousands of employees and files, and you get the idea. Each file's footprint may be small, but when replicated and spread across the spectrum of daily business, the issue of duplicate data can become a serious problem.


E-Mail Preservation

E-mail is another tricky area. Can an agency realistically archive all e-mails? Should it? If so, backups may include huge amounts of spam that has slipped through firewall precautions, newsletters, personal dialogs and files, etc. Are policies and procedures in place requiring users to self-manage their e-mail archives, or is that an agency-enforced process?

A wide variety exists in government processes regarding e-mail limits and management. On one end of the spectrum, Stanislaus County, Calif., doesn't impose size limits on e-mail folders at all; instead, a retention time of 45 days is implemented. Any items in the e-mail system after 45 days are automatically deleted. If a longer retention time is required, users are expected to archive those e-mails on other file servers.

At the opposing end of the spectrum, Alabama offers four levels of personal mailboxes at different costs, based strictly on physical size. The largest mailbox offered is 2 GB. Burns stresses that e-mail is a messaging system and not a file-storage system. The responsibility of deciding which e-mails need to be archived on a different network file storage location falls on the particular agency and user, along with help

from IT staff.


Storing Chaos

Everyone has a junk drawer in the kitchen. We toss spare pens, matchbooks, slips of paper containing faded phone numbers and a multitude of other odds and ends into it. For some, the junk drawer eventually becomes so full that they start using a second drawer. We don't dare throw something away on the off chance that it one day may become critical. But what happens when a crisis arises and you need to find that fading phone number? Which drawer is it in? Who remembers what it looked like? Is it still readable? The little slip of blue paper can no longer be found buried within the mess, and even if it could, it's in such bad shape that it's no longer readable. We realize that, after months become years, some of the items tossed haphazardly into the drawer are of little to no value.

The same concept applies to data storage. If we're allowed to save everything and anything, the amount of junk tossed into our directories will accumulate into chaos over time. Having no limits can create cobwebs that will often never be cleaned out unless management forces the user to do so.

It's also far too easy for users to take advantage of unlimited storage at the workplace by saving personal photos, correspondence, video, etc. The lack of limits can propagate misuse of agency resources for personal use. Staff members are cautious with what they choose to save or store because they fear Big Brother might be watching. Remove that threat and people may be tempted to save or store whatever they please. Not only does this tie up valuable resources, but it may also open the employer to liability.


When Changes are Needed

The cost of physical storage is decreasing, storage needs are increasing and a wider variety of options are available. When is the right time to make a change in the way an agency does business?

Stephanie Balaouras, senior analyst of Forrester Research, said, "according to interviews with enterprise end-users, data growth coupled with shrinking backup windows is the main driver behind adoption of disk-based data protection."

Here are some points to consider:

  • Costs of performing a manual backup. How much staff time (and cost) is eaten up doing manual backups and data restores?
  • Length of the backup window. How often is the backup process aborted because the window of time has expired?
  • Backup and recovery failure. How often does the tape backup fail?
  • Other processes affected. Are other processes or projects placed on hold so that the backup can be performed?
  • Legal requirements. Can your organization meet audit needs or discovery requests?


Storage Solutions

"There was a time when we did things in IT because they were cool, but today it's very unusual to see anything happen that doesn't really give you bottom-line value to the business," said Michael Krieger, vice president of the Market Experts Group for Ziff Davis Enterprise.

As agencies deal with this complex issue, they are considering the implementation of a strict data/e-mail retention policy and operating procedures. Proactively eliminating waste can significantly improve the overall data storage, backup and restoration process. According to a survey from Byte and Switch, users report that the biggest problem with an entity's data protection plan is simply having too much data to handle within an "absurd" backup window.

Much of the public sector is also considering adopting a data deduplication solution. Deduplication technology compares files and only stores the bytes that change. ExaGrid Systems Inc. claims that only 2 percent of bytes typically change from backup to backup. Though this technology is expensive, 59 percent of those surveyed by said deduplication is the top storage-networking technology under consideration.

Even as the economy slows, government must continue. Using taxpayer money requires that the public sector analyze and implement best practices for business, along with examining the need for and costs associated with unlimited data storage.

Leslie Friesen Contributing Writer