For the Record

The digital revolution raises difficult new questions about how to maintain a complete account of government activities

by / May 27, 2005
Reprinted with permission from the May issue of Government Technology's Public CIO

When future generations look back on our nation's history, the 25 years leading to the present day will appear a bit sketchy, said Timothy Slavin, director of the Delaware Public Archives and president of the National Association of Government Archives and Records Administrators (NAGARA).

"[This period] may not be the Dark Ages, but it may be the dim ages when it comes to accessing certain kinds of public records," Slavin said. That's because, as governments increasingly rely on electronic information systems, the ability to produce digital records has far outstripped the ability to manage and preserve them.

"For years, we've had our records retention policies based on the understanding that we do everything on paper," said Matt Miszewski, CIO of Wisconsin. "Unfortunately simply because we've moved to a better, more efficient technology, we're losing some of the record about what we did."

Federal, state and local laws require governments to retain records of their activities and make them available to people authorized to view them. For everyday business matters, the required retention period may be a couple of years or a couple of decades. For the small percentage of records deemed historically significant, the retention period is virtually forever.

More and more of these records are created and maintained in digital formats. In the past, documents and transactions created on computer systems routinely yielded printouts that could serve as permanent records of a government's activity. Today, however, a growing number of records are born digital and never pass through a printer, and many paper documents are scanned and managed in digital form. Government officials charged with maintaining the public record have started to appreciate that life in the Digital Age poses new challenges. Gradually governments are starting to develop strategies to overcome them.

Media Migration
One question about how to preserve electronic records centers on choosing a physical storage medium. Anyone who tries using a current-day computer to view data stored years ago on a 5.25-inch floppy disk will understand why records managers must keep this issue in mind.

The market offers good media for preserving electronic records, Slavin said. "The trick is, people need to understand that electronic records need to be migrated onto new media." Data storage must keep up with evolving technologies. "Costs don't go away when [the records] are written and taken offline or near-line. The costs keep incurring."

Those costs are not prohibitive, however, said Ken Thibodeau, program director of the Electronic Records Archives (ERA) program at the National Archives and Records Administration (NARA). "Historically costs of digital media decrease by 50 percent and capacity increases by 50 percent over a period of about two and a half years. By migrating to newer media, you can actually cut your costs in half." But the migration must be done efficiently, he said.

Even so, with digital records growing in volume and complexity, storing the data is a serious problem. Robert Horton, Minnesota's state archivist, recalled copies of maps he received from the Minnesota Department of Transportation (Mn/DOT). "They were enormous TIFF images. We received 3,600 files on approximately 300 DVDs; the total storage came to 1.2 terabytes. The first time we tried loading that information on a server, it took approximately 50 minutes per DVD." Eventually the staff reduced the process to 10 minutes per DVD, he said. "But that was one set of records from one agency. Multiply that times all the people I could conceivably be dealing with. The system would break down very quickly."

Although managing physical storage media takes time and money, it doesn't pose fundamental questions about survival of the records, Thibodeau said. "You can migrate bit streams with no loss, or no alteration, to newer media." A much trickier matter is how to ensure that people 10, 20 or 100 years from now can access records created with today's software.

"The format issue is much more complex because it's not driven by the people who need the information," Thibodeau said. "It's driven by the people who want to sell new versions of software." When vendors update applications, they don't necessarily think about preserving the integrity of old records. For example, a document created 20 years ago might be readable in a current word processing program, he said. But if the technology used to create footnotes has changed over the years, footnotes in that document might not appear when viewed in the new application.

"A government archive can't be a museum of old computer systems," retained for the sake of accessing older records, Slavin said. "You have to have some kind of software-independent format in which to store these things, or if you are software-dependent, it has to be a system that's widely recognized and available."

Along with deciding how to save electronic records, governments must decide which records to retain. The same question applies to paper, but policies already exist for managing paper records. Many kinds of electronic records -- such as e-mail messages and Web pages -- have no analogy in the paper world; governments must create new policies about which to keep and which to destroy.

Save Everything?
Minneapolis, for example, is engaged in a long-term project to revise its records management policies to cover electronic and paper records. Gradually the city is developing retention schedules for both the enterprise and individual departments. Once state officials approve a retention schedule, the city can destroy records that have reached the end of their life cycles, said Craig Steiner, records manager for Minneapolis. "In the meantime, we're maintaining all electronic records that have not appeared on a retention schedule."

In some cases, fear of losing important records has driven over-archiving of electronic content, said Doug Robinson, executive director of the National Association of State Chief Information Officers. With storage prices falling, "IT people say, 'Why worry about it?'" But while it may be easier in the short run to save everything rather than to impose order, it's important to develop a classification system and archive only critical documents, he said.

Even though storage prices are lower than ever, the volume of electronic records has soared so much that total costs are rising, Miszewski said. And as archives expand -- particularly if they expand without an enterprisewide standard for formats and records management systems -- retrieving information can be extremely burdensome, he said. "A simple request to give me all the documents related to a specific person can turn into a three-month endeavor," which carries costs of its own.

Two other challenges records managers face in the Digital Age involve cultural change. For one thing, end-users must learn to treat the electronic records they create or receive as records. "We have to figure out ways of training users and incorporating that knowledge of record keeping into everyday functions," Slavin said.

Now that all end-users can generate records at the desktop, governments must figure out how to involve everyone in records management, said Horton. "They're going to be naturally resistant to that. You can send out a policy that says, 'Make sure you treat all your e-mail messages as records,' but if you're getting 200 e-mail messages a day and you don't have a system in place to facilitate storage of e-mail, some people will delete all of them. Other people will save every single one." When a government develops records management policies, the policies must be practical to implement, he said.

The other cultural imperative is to make government IT departments build records management functions into applications from the start, but IT professionals aren't used to thinking that way, said Miszewski. "They're trying to accomplish business goals, and sometimes it's secondary that they're considering records retention issues. Changing that culture is extremely difficult, especially in a time of budget constraints." But when they understand that efficient records management helps cut storage and retrieval costs, IT managers will rally to the cause, he said.

NARA's Initiatives
All levels of governments are starting to implement policies and launch new initiatives to meet the challenges of digital records management. At NARA, the ERA program is working to acquire a system to enable the National Archives and the presidential libraries to preserve and retrieve any kind of electronic record, Thibodeau said.

As part of the ERA program, NARA is developing new standards and mechanisms to govern the way federal agencies manage electronic records, some of which eventually will pass to the National Archives, Thibodeau said. This will ensure that NARA can obtain all the necessary records in formats that allow them to endure through the years and maintain their authenticity.

The move to electronic records requires a fundamental shift in governing records management, Thibodeau said. Traditionally records managers tried to apply standards from the paper world to electronic records. They categorized them in the same hierarchical fashion and stored them in the digital equivalent of filing cabinets, separate from the information systems in which they are created and used.

This is not necessarily the best approach, according to Thibodeau. NARA is looking at alternative ways to integrate records management functions right into business applications. "We're looking at [defining] records management services that are nuggets of software that can be implemented in the computer systems and do things like point to what's in this database that's actually a record," he said. "And maybe map it across several tables, and allow you to control that. Or maybe another service that allows you to destroy a record when it's time -- just say, 'Go into my system and destroy all the records eligible under this authority.'"

NARA is also exploring suitable data formats for records preservation. Since the most effective format for preservation often isn't the format that meets agencies' daily business needs, archivists must carefully impose standards, Thibodeau said. The solution might be to specify a "transfer format" to which agencies could convert records when it came time to move them to an archive for long-term preservation, he said.

Extensible markup language (XML) is probably the best known format in this category, and it's one archivists often mention as a standard for records preservation. "Even if all the software today were to disappear, you have to assume a computer in the future could read XML tags and XML schema the same way computers today can read plain ASCII," Thibodeau said.

Persistent Archives
Minnesota is working with Michigan, Ohio, Kentucky, California, Kansas, several university archives and the San Diego Supercomputer Center to tackle the problem of storing huge records such as Mn/DOT's map files. Funded by the National Historical Publications and Records Commission, the Persistent Archives Testbed (PAT) is testing the use of data grid technology to store very large electronic archives. Grid technology uses the Internet and an application called the Storage Resource Broker to distribute archival responsibility over a network, Horton said. In its pilot project, Minnesota's state archive at the Minnesota Historical Society gathers the information, describes it by attaching standardized metadata, and then using grid technology, stores it in San Diego, where the available storage capacity far surpasses what Minnesota has. With records stored this way, Minnesota won't need to worry about accessing data from DVDs or other unwieldy offline storage media, or periodically moving stored data to newer media.

In Wisconsin, an administrative rule established in 2001 sets enterprisewide policies for electronic records management, Miszewski said. The state also published "a business-oriented primer" to help employees interpret the rule, so they understand what does and does not need to be retained, he said.

The state has been getting stakeholders to specify their records retention requirements and is preparing to procure an information system for managing records across the entire state government. It's important to take an enterprise approach because Wisconsin's state workers often move from one department to another, Miszewski said. "We'd like to make sure the records management system is the same across those functional silos."

As part of a project to consolidate the state government's e-mail systems, Wisconsin is developing a draft standard to govern how users should save e-mails as public records. Creating policies for an application as widely and frequently used as e-mail is complicated. "And we're not done," Miszewski said. "There is still a significant amount of debate going back and forth among different departments that have really different issues with regard to e-mail retention."

In Delaware, a "quick and dirty" study of e-mail conducted along with two other states revealed that only 2 percent to 5 percent of the messages in government users' inboxes meet the criteria for records that need to be maintained for long periods, Slavin said. "But the problem we have is that it's easier to keep [messages] electronically -- you just build a bigger mailbox."

Governments must train users to manage records like these at the point of use, Slavin said. Records management systems that can automate some of this task for users are starting to emerge, he said.

Building It In Up Front
To this end, Delaware published a series of guidelines to help ensure that when government agencies develop applications, they build records management functions into them. "Records management used to be done primarily after records creation," Slavin said. "Now we're beginning to see records management done prior to records creation."

Building records management into applications up front is a priority for Minneapolis as well. Together with the city's Department of Business Information Services, the Records Management department developed an initiative called Enterprise Information Management (EIM). Among the policies developed under EIM is one stipulating that every time a city department develops a new information system, part of the project budget must be devoted to records management concerns.

So when the City Attorney's Office developed a new case management information system last year, money was earmarked for information management requirements, which includes the retention schedule and record keeping requirements, Steiner said.

City departments will also have to add information management requirements as they update their five-year business plans. "Not only do we want to hit departments when they're developing a new system, we want them to be strategically planning for how they're going to meet their basic information management requirements," Steiner said.

Initiatives like these could soon pull governments out of the dim ages and back into the light. It's an urgent mission, Miszewski observed. "The records being lost are lost forever. The ones being over-saved are incomprehensible and unusable." The result, he said, "is simply unacceptable moving forward."
Merrill Douglas Contributing Writer
Platforms & Programs