Reprinted with permission from the May issue of Government Technology's Public CIO
When future generations look back on our nation's history, the 25 years leading to the present day will appear a bit sketchy, said Timothy Slavin, director of the Delaware Public Archives and president of the National Association of Government Archives and Records Administrators (NAGARA).
"[This period] may not be the Dark Ages, but it may be the dim ages when it comes to accessing certain kinds of public records," Slavin said. That's because, as governments increasingly rely on electronic information systems, the ability to produce digital records has far outstripped the ability to manage and preserve them.
"For years, we've had our records retention policies based on the understanding that we do everything on paper," said Matt Miszewski, CIO of Wisconsin. "Unfortunately simply because we've moved to a better, more efficient technology, we're losing some of the record about what we did."
Federal, state and local laws require governments to retain records of their activities and make them available to people authorized to view them. For everyday business matters, the required retention period may be a couple of years or a couple of decades. For the small percentage of records deemed historically significant, the retention period is virtually forever.
More and more of these records are created and maintained in digital formats. In the past, documents and transactions created on computer systems routinely yielded printouts that could serve as permanent records of a government's activity. Today, however, a growing number of records are born digital and never pass through a printer, and many paper documents are scanned and managed in digital form. Government officials charged with maintaining the public record have started to appreciate that life in the Digital Age poses new challenges. Gradually governments are starting to develop strategies to overcome them.
One question about how to preserve electronic records centers on choosing a physical storage medium. Anyone who tries using a current-day computer to view data stored years ago on a 5.25-inch floppy disk will understand why records managers must keep this issue in mind.
The market offers good media for preserving electronic records, Slavin said. "The trick is, people need to understand that electronic records need to be migrated onto new media." Data storage must keep up with evolving technologies. "Costs don't go away when [the records] are written and taken offline or near-line. The costs keep incurring."
Those costs are not prohibitive, however, said Ken Thibodeau, program director of the Electronic Records Archives (ERA) program at the National Archives and Records Administration (NARA). "Historically costs of digital media decrease by 50 percent and capacity increases by 50 percent over a period of about two and a half years. By migrating to newer media, you can actually cut your costs in half." But the migration must be done efficiently, he said.
Even so, with digital records growing in volume and complexity, storing the data is a serious problem. Robert Horton, Minnesota's state archivist, recalled copies of maps he received from the Minnesota Department of Transportation (Mn/DOT). "They were enormous TIFF images. We received 3,600 files on approximately 300 DVDs; the total storage came to 1.2 terabytes. The first time we tried loading that information on a server, it took approximately 50 minutes per DVD." Eventually the staff reduced the process to 10 minutes per DVD, he said. "But that was one set of records from one agency. Multiply that times all the people I could conceivably be dealing with. The system would break down very quickly."
Although managing physical storage media takes time and money, it doesn't pose fundamental questions about survival of the records, Thibodeau said. "You can migrate bit streams with no loss, or no alteration,