Formatting for the Future

A second state looks to switch to open formats for state documents. Will the fight get as ugly as it did in Massachusetts?

by / July 31, 2006

Massachusetts leaders had a seemingly simple idea: Government documents produced by word processing, spreadsheet, database or presentation software belong to the public record, and therefore, should be created and stored in open formats that aren't tied to a particular program.

Massachusetts reasoned that storing electronic documents in proprietary formats might create situations where documents were held hostage by those formats. Important public documents could be rendered all but inaccessible when the applications that created them went out of style.

The Massachusetts Information Technology Division (ITD) explained the desire to move to an open data format on the frequently asked questions page of its Web site:

"Ease of access to electronic records created in proprietary formats is limited in time. Once the proprietary vendor abandons a particular version of an application or format, documents created and formatted in those applications and formats may become inaccessible to all readers. The proprietary formats supported by our current office applications may place a permanent lock on future access."

This is where Massachusetts' simple idea got complicated.

To create government documents using open data formats, the agencies making those documents must use nonproprietary software tools. That means a massive shift away from the dominant desktop software suite -- Microsoft Office.

The threat of that change set off a pitched battle that led to the resignation of state CIO Peter Quinn. And with the dust still not settled in Massachusetts, another state -- Minnesota -- is exploring a move to open document standards.

Microsoft -- intent on protecting its market position -- announced in late 2005 that it would submit its Office Open extensible markup language (XML) format to a European standards body for review, a key step toward getting Open XML certified as an open standard.

To further muddy the waters, experts warn that the lack of a clear definition for open standards complicates states' move toward electronic document formats that will be widely accessible over the long term.

Preserving Documents
Government IT leaders face a document dilemma -- especially with so much government information being created, shared and stored electronically.

From a government's perspective, the format of an electronic document is a significant issue. Proprietary formats are quite needy -- they refuse to work with any other program besides the one that created them; they require a current, running copy of that program; and they need that program to be installed on a computer and operating system that can run the program.

This creates challenges for state and local governments that must respond to information requests from a wide spectrum of constituents who want to review public documents associated with a range of government projects or services programs.

Massachusetts officials contend that government documents should be created in a format readable by various applications, noting that some constituents can't afford a computer with an up-to-date version of the Windows operating system and Office suite.

Another problem with proprietary formats is that an electronic document created by agency X may not be compatible with agency Y's computer system, a situation that hinders information sharing among agencies and levels of government.

Massachusetts' 2005 proposal for dealing with these two problems whipped up quite a disturbance indeed. But creating documents based on open data formats isn't just about the documents themselves.

In early 2005, the Massachusetts ITD announced it would begin work on an Enterprise Technical Reference Model (ETRM) for the state. The rationale for creating the ETRM was straightforward: to provide an "architectural framework used to identify the standards, specifications and technologies that support the commonwealth's computing environment," according to the ITD's Web site.

But the ITD's commitment to several "open initiatives" as overarching themes of the ETRM sparked a political skirmish that resulted in Quinn submitting his resignation letter in December 2005 -- a scant three months after the ETRM was finalized.

As part of the move to open standards, the ETRM directed executive-level agencies to create text, spreadsheet and presentation documents in the OpenDocument format. OpenDocument, short for the OASIS Open Document Format for Office Applications, is a file format for saving, editing and exchanging office documents, such as text documents, spreadsheets, databases and presentations.

Those involved, including Quinn and new Massachusetts CIO Louis Gutierrez, said open standards are the future, and that governments must make the move to standards-based computing architectures to better share information with each other.

From the beginning, however, the ETRM's open standards ideology was cast as the ITD throwing down the gauntlet against vendors that sell systems and software based on proprietary formats. In media coverage, the OpenDocument directive was cited as proof of the ITD's intentions -- to comply with the directive, executive-level agencies would be forced to use office applications that support the format. This would freeze Microsoft out because its suite of office applications doesn't support OpenDocument.

The ITD's OpenDocument decision set in motion a series of political events.

Microsoft lobbied Massachusetts lawmakers against adopting the OpenDocument format, according to news reports, and sent a 15-page letter to the ITD and a copy to Gov. Mitt Romney's office.

In November 2005, the chairman of a Massachusetts Senate committee conducted hearings to determine whether Quinn and the ITD had authority to promulgate policy for executive branch agencies. Massachusetts' secretary of state also questioned the ITD's ability to set policy for state records.

Then in late November, The Boston Globe ran a story questioning Quinn's travels to several technology conferences. The Globe said it raised the issue because some of the conferences were sponsored, in part, by IT companies that stood to benefit from the OpenDocument decision.

Massachusetts investigated Quinn's travels, later exonerating him of all allegations of impropriety.

The fight lives on even now. In late July 2006, state Sen. Marc Pacheco, chairman of the Senate Committee on Post Audit and Oversight, released a report revealing the results of a monthlong investigation into the ITD's OpenDocument directive. The report, Open Standards, Closed Government, sharply criticized the ITD's actions.

"ITD did not have the statutory authority to issue provisions in the ETRM relating to public records management," according to the report. "The Secretary of the Commonwealth and the Records Conservation Board both have jurisdiction over public records in Massachusetts and did not approve the ETRM as required by Massachusetts statute."

The report also urged the ITD to delay the ETRM's implementation date of Jan. 1, 2007.

The governor's office fired back. Romney's spokesman, Felix Browne, was quoted in media coverage of the report as saying, "Senator Pacheco is wrong on the facts and wrong on the law. We are committed to an open standards approach that fully takes into account all accessibility, cost and statutory requirements."

Standards Are Good For You
Gutierrez, who directs the Massachusetts ITD, inherited the job of moving forward on the ETRM. He said the target date for the first group of Massachusetts agencies to start using office productivity applications that support the OpenDocument format is January 2007.

Gutierrez said the ITD is piloting three ETRM-compliant office suites -- OpenOffice, StarOffice and IBM's Workplace Managed Client.

Gutierrez took the job despite the well publicized OpenDocument conflict, which might have scared off many prospective candidates.

"I didn't come for the conflict -- I came for the principle," he said. "The principle is that the IT division is in pursuit of a standard. We want to approach it the most intelligent way possible. It's not a discriminatory approach in terms of any vendor or vendor class."

Gutierrez acknowledges the history of the conflict -- on one side is a new breed of open source applications that use a new document format, and on the other is an entrenched set of applications that don't -- but said that history created a debate that was incorrectly centered on applications.

"The standard is about the format, not the application versus application," he said. "It's really a pointer toward something that Gary Edwards [founder and president of the OpenDocument Foundation] is very compelling in his discussion of -- we are heading toward a document-centric world and moving away from an application-centric world. It's going to be much more important how we structure, store and have workflow with standardized document forms than whether I'm using this or that version of this or that office productivity app."

This movement is inevitable, even for states that aren't yet incorporating the OpenDocument format, Gutierrez continued, because those states will have to contend with the coming conversion from binary formats to XML-based document formats as enterprises of all stripes see the wisdom of platforms and documents built on open standards.

"You look at this and you say, 'Right now, we all rely on the Internet with its TCP/IP standard, and, boy, it's just like running water,'" he said. "That wasn't always the case, and in the early days, we used to wander around with different networking schemes, topologies and means of trafficking over the wire. Now we've come to a standard. It's not a vendor-specific standard, and it just works."

It's far too early to tell whether documents will achieve that level of uniformity, Gutierrez said, though he predicts people will gradually make peace with the notions of transparent and standard-based document storage.

"The direction of this standard is almost self-evident," he said. "It really is compellingly useful and interesting to have governments start to traffic in open, standardized, XML-based document formats.

"To the extent we think of this as enabling the documents, that any given person at their desk saves, to better interoperate in broad-scale workflow, archiving and indexing -- that's when things get exciting and interesting," Gutierrez continued. "That's really the ultimate play here."

He said he was pleasantly surprised by a May 2006 announcement from the OpenDocument Foundation that it was working on a plug-in to let Microsoft Office users open, edit and save files based on the OpenDocument format. The plug-in also would allow files to be translated between OpenDocument format, XML and formats -- such as .doc, .xls and .ppt -- used by Microsoft.

"If that pans out," Gutierrez added, "it opens up interesting implementation possibilities."

Adapting to Open Format
Gutierrez might get some help from Microsoft itself.

The company did an about-face of sorts when in November 2005 it said it would submit its Office Open XML formats to the European Computer Manufacturers' Association (Ecma) International, a European standards body, for review. The process will take approximately one year.

A technical committee within Ecma -- sponsored by Microsoft and companies such as Apple, BP, Essilor, Intel, NextPage, StatOil and Toshiba -- will review the Open XML format's documentation, or the narrative description of how the code behind Open XML works, and how it interfaces with hardware and other software. If Ecma approves the standard, Microsoft said it would present the standard to the International Organization for Standardization (ISO) to seek international status for the Open XML standard.

Assuming everything goes Microsoft's way with these two standards bodies, Massachusetts might be able to stick with its Windows boxes loaded with Office software, instead of a migrating to a completely new desktop productivity tools.

Alan Yates, general manager for business strategy in Microsoft's Information Worker Group, said the company never had a problem with the Massachusetts ETRM itself. The software giant did object, however, to a lack of clarity in the state's OpenDocument directive, he said.

"The policy was written to favor open standards, and then, at the time, it picked out the OpenDocument format as the only thing that met their criteria, but the criteria weren't very clear," Yates explained. "Then the criteria created an exception for [Adobe's] PDF [portable document format] at the same time, which created more confusion. PDF wasn't a standard at the time that they chose it, and then later on it became sanctioned by a standards organization.

"At this point, the secretary of Administration and Finance has said that when Microsoft Open XML reaches Ecma standardization, it's likely to qualify -- and that they like the direction we've set," Yates said.

Microsoft believes government can benefit from having a choice of standards, he said, because of the competition between standards vying for a government's attention -- but nobody benefits by being forced to move to one particular standard.

Minnesota Eyes Open Standards
In late March 2006, Minnesota state Rep. Paul Thissen introduced HF 3971 in the state Legislature. The bill instructs Minnesota's Enterprise Technology Office to "assist state agencies to avoid the purchase or creation of data processing devices or systems that do not comply with open standards for the accessing, storing or transferring of data."

The bill didn't make much progress during the legislative session, Thissen said, but that was by design.

"Part of the plan this year was to get the idea out there and circulating," he explained. "It usually takes a little time for these things to germinate and come to fruition. Next year we'll make a much bigger effort, and we'll try to educate folks during the off session here."

Thissen said it's difficult to say whether his colleagues have formed substantive policy positions on the bill, especially this early on, and acknowledged that he learned two important lessons from observing the Massachusetts' situation. First, educate as many people as possible on what the bill means.

"People, generally, in government have vague ideas of what open standards and open source are, but they don't understand specifically what the implications of those things are," Thissen said. "An education piece is important, which is part of the reason we wanted to raise it this year, and we're doing some work with local newspapers as well to write opinion pieces to educate the general public on how important this is."

Though the practice of writing software and building hardware to open standards isn't new, the concepts and the language describing them is absolutely foreign to the average person and the average legislator, he said.

The second lesson is the importance of giving a clear definition of open standards that's also backed by outside parties.

"We're working hard to figure out what that is," Thissen said. "Because once there's an accepted definition -- one that's not just a Minnesota government definition, but one that would be accepted by private industry in Minnesota, and more generally, across the country -- people get more comfort from that."

The bill's language falls short of mandating that agencies make the switch to hardware and software based on open standards, but doesn't necessarily give agencies a free pass.

"It's more than an option, and a little less than a mandate," Thissen said. "There would be a strong preference for the use of open standards, unless the nature of the project precludes the use of an open data format. There has to be a determination that an open standard format is not going to work."

The bill also would require review of existing formats every four years to allow the state government to adopt future formats that adhere to open standards. In this way, Thissen said, Minnesota gets the flexibility to replace IT systems or hardware based on proprietary formats as open formats evolve.

Finally the bill asks state agencies to make sure the documentation on open data formats is accessible to anyone, he said, so other entities inside the state and beyond can use that documentation.

Necessary Spark
Thissen said the legislation is needed because state agencies might not otherwise be willing to take a chance on standards-based software or hardware.

"We want to jumpstart the move to open standards," Thissen said. "I think that's necessary because it's just easier to go to a proprietary standard, and like private business, public agencies don't want to be Massachusetts. They don't want to be out front. They want to go with the safe choice.

"I believe open standards formats are a safe choice," he continued. "But if there's a legislative directive to go in that direction, then state agencies are obviously going to feel much better about the choice. It may be the thing that gives them the comfort level to move in that direction."

Fortuitously Thissen's open data format legislation could play a role in a larger statewide transformation program. Gov. Tim Pawlenty announced his "Drive to Excellence" initiative in April 2005 to change how Minnesota state government operates, according to the governor's Web site, by thinking of state government as a single enterprise that serves all citizens, instead of a collection of independent agencies serving their own constituencies.

"One of the pieces of the Drive to Excellence is IT consolidation," Thissen said. "As we go through that process, we want to make sure we're thinking about where we're going to end up on the far end, in terms of open standards. The bill fits, in some sense, very well with this idea of the Drive to Excellence -- which is to remove some of those barriers and silos."

Thissen said he and his staff met with members of the governor's office to discuss the bill and the Drive to Excellence. The lawmaker described the talks as preliminary, but acknowledged that he and his staff will have to work closely with the governor's office during the next legislative session to address concerns with the bill's language.

Sticking to One Definition
More states moving toward hardware and software based on open standards could trigger new problems if those standards aren't consistent.

"I didn't think that the specific wording of the [Minnesota] amendment was very well informed," said Andrew Updegrove, a lawyer with GesmerUpdegrove LLC, who specializes in high-tech legal affairs. "Superficially it goes to a good place. But as a practical matter it wouldn't encourage, from a marketplace point of view, very good results.

"The value that standards can offer is totally dependent on lots and lots of people adopting the standards," Updegrove continued. "What if you gave a standard and nobody came? Well, you'd have a piece of paper that would be useless."

The problem, Updegrove explained, is that if every state sets its own eclectic definition of what an open standard is, then nothing has changed from a marketplace where there are no standards and every buyer has a custom specification.

"Instead of trying to adopt and endorse an existing definition of open standards, [Minnesota] decided to create its own," he said. "That's the sense in which it's counterproductive. While it overlapped with what prevailing definitions of open standards are, it was an approximate fit in many cases.

"I should hasten to add that there's quite a debate right now over what open standards should mean," he continued. "That's because there's a whole bunch of different areas converging."

Updegrove said a well equipped cell phone offers an excellent example -- such a phone might offer a camera, a Web browser, PDA software and games, as well as actual telecommunications capabilities, and all of those features are built to particular standards.

Those standards come from different evolutionary paths, he said, and when they all meet in one device, things get dicey. If one standard for one piece of software has one sort of patent attached to it, and another standard for another piece of software has a completely different and contradictory patent attached to it, then is the device really built to open standards?

As a result, "open standards" means different things to different to people.

"There's rather a clash today between the open source community and what they think the definition of open standards should be, and the proprietary software vendors and what they think the definition of open standards should be," Updegrove said.

Even so, he added, the fact that open standards seemingly attract attention from lawmakers is encouraging.

"It's very interesting that this kind of thing can be out there, even as a topic," he said. "Legislatures, historically, haven't been very engaged in something this technical. For a state government to get intrigued is kind of intriguing."

The Massachusetts Experiment
The potential for creating uncertainty by adopting differing definitions of open standards is something states must pay attention to, said Quinn, now senior vice president of Bisys Group, which provides IT outsourcing services to the financial services industry.

"If this many people create this many definitions, it will become extremely difficult for vendors and standards bodies to actually opine on something that would make any sense to the universal populace," Quinn said. "I would tend to err on the side of what you see coming out of the recognized standards bodies on what constitutes an open standard and how it is that they ultimately ratify it."

Quinn said he's still surprised that the Massachusetts open document standard caused such a ruckus, especially since the prime motivation was simply to create documents that would be available to anybody, now and in the future.

"The mainstream business press completely ignores the issue," he said. "It frustrates me, from the standpoint that what does get written are snippets. There are no real in-depth articles that get written about this and the importance of historical documents and access to those documents by future generations.

"This is a significant issue that goes well beyond governments," he continued. "If the general populace thought their kids or grandkids couldn't get at a historical document -- like you can walk into a state archive or state library today -- I think they'd be pretty incensed. At least I'd like to think they'd be incensed."
Shane Peterson Associate Editor