Transparency vs. Privacy and Security: What's an Agency to Do?

Means opening up the records, information and processes of the agency to timely public inspection and, further, opening up communication lines for the public to talk back.

by / December 29, 2008 0

Illustration: Opening government and protecting privacy are initiatives that can often travel in opposite directions.

"Transparency" is an up-and-coming buzzword that is finding its way into the national conversation at the federal, state and local levels. Its continued rise to prominence is pretty well assured when the new administration takes office because President-Elect Obama has been associated with federal transparency initiatives for years and has, at least for some federal agency CIOs, made transparency an important part of the transition dialog.

But what does transparency mean for the agency head or IT manager who has been instructed to make his or her agency "more transparent"? What are some of the key issues and architectural considerations that need to be addressed?

Government Transparency

First, some background. Although "transparency" as a term has earned recent cachet, the debate about it is quite old, often found in discussions about government openness or implicit in discussions about public disclosure policies and the Freedom of Information Act. And while its current use focuses on opening up government processes, it has also been used as a political tool to bring about changes in the private sector. In their book Full Disclosure: The Perils and Promise of Transparency, authors Fung, Graham and Weil explore the public policy implications of transparency generally and identify "targeted transparency" as a tool available to federal, state and local leaders to help redress wrongs and increase public safety. They cite federal mandates for public disclosure of automobile rollover risks as an example of its use. In that case, the federal government required disclosure by private companies of specified product safety data with the intent of helping consumers make more informed choices and to leverage natural market forces to bring about long-term improvements. It worked.

But for the agency head, transparency means something different -- it means opening up the records, information and processes of the agency to timely public inspection and, further, opening up communication lines for the public to talk back. In other words, we're now talking about providing a means for them to comment on what they see or would like to see.

It is this kind of transparency that President-Elect Obama helped to champion by cosponsoring the Federal Funding Accountability and Transparency Act of 2006 and more recently co-sponsoring the Strengthening Transparency and Accountability in Federal Spending Act of 2008 (S.3077). The president-elect's description of the 2006 Act is that it "...created the public Web site, makes information about nearly all federal grants, contracts, loans and other financial assistance available to the public in a regularly updated, user-friendly and searchable format. The Web site includes the names of entities receiving federal awards, the amounts of the awards, information on the awards including transaction types, funding agencies, location and other information."

In his floor speech on June 3, 2008 introducing the 2008 bill, Senator Obama commented that the new bill "...will improve government transparency and give the American people greater tools to track and monitor nearly $2 trillion of Government spending on contracts, grants and other forms of assistance."

Privacy, Security and Other Contradictions

While transparency sounds like a good idea and has met with successes, it is not without its challenges and even contradictions. For example, during the same period that transparency has become a common talking point, so too have mandates to protect personal privacy information (PII): opening government and protecting privacy are initiatives that can often travel in opposite directions.

The following are some issues that surface immediately when a transparency program is

initiated. A tremendous amount of information is available about each -- the following serves to simply introduce the concept and its relationship to transparency efforts.


Today, there's more concern than ever about privacy due to the advent of identity theft, stalking and other potential abuses of online data. As a result, many laws have been passed in the U.S. and elsewhere, forbidding display on Web sites of "personally identifiable information" -- data that makes it possible to identify a specific individual. While this may sound simple, it can rapidly become quite complex because combinations of otherwise innocuous data can sometimes be used to identify an individual. That is, one datum may not in itself be PII (for example, a birth date), but if a site includes a birth date, the person's sex and the person's home ZIP code, the combination of these three data enable individual identification in more than 80 percent of the cases. Yet none of these three facts, in themselves, constitutes PII. Drafting legislation or building business logic to accommodate this kind of fuzzy situation can be challenging.

Then there's the problem of old records. Mandating that new documents don't contain PII is one issue; the greater challenge is what to do about old documents. In many cases, old paper documents have been scanned to images which means there is no simple, fully reliable method for electronically reviewing the document to identify PII information. So, reliably cleaning old documents can be time consuming and expensive.

Protecting privacy also has its counter-arguments. It is generally true, for example, that the value of a document is inversely proportional to the quantity of data removed from it: the more data removed, the fewer legitimate uses the document is likely to have. For example, a death certificate without a signature is worthless as a legal document but displaying signed death certificates online could facilitate certain crimes by providing a template of a person's signature.

Trade Secrets

Many agencies have access to what amounts to proprietary trade secret information about private companies. As those agencies open their doors to greater public inspection, they need to make determinations about how to protect trade secrets. While it would be clear that any federal agency having access to the formula for Coke would have to take actions to protect that information, other issues are less clear. For example, the Strengthening Transparency and Accountability in Federal Spending Act of 2008 includes a provision that will require agencies to post online facsimiles of and text-searchable versions of all contracts (in addition to the original RFP, award, and other related information). To the extent that the contract contains detailed financial or other information related to the awardee, it is possible that trade secrets could be revealed.


Security has been the watchword for government in recent years, particularly since 9/11. Security and transparency are another contradictory pair: the safest data is that information that no one knows and that can never be relayed. It also happens to be the most useless, though in some cases that might be the point. Still, no matter where you stand on government secrets, few would ever argue for eliminating all secrets. So, transparency needs to be linked to a keen understanding of what is and is not secure. For some agencies, such as Housing or Land Management, security per se may be a relatively minor issue (which they would make up for in Privacy and Trade Secret concerns), but for many agencies (such as the Department of Defense or the Nuclear Regulatory Commission), it is a core concern.

When it comes to developing systems, security can't be overlaid as an afterthought -- it needs to be baked into any solution from the beginning while making sure that it doesn't turn into the sole, dominant


Employee Relations

A particular type of transparency has been tried in several states which highlights how transparency can have an impact on the agency workforce. In those states, the names and salaries of all agency employees have been made public. Although this is an interesting approach to opening government, it can also result in complaints from government employees who don't necessarily want their paychecks shared with the world any more than the rest of us would.

It's Not the Technology

As with many things IT, the most challenging aspects of trying to become more transparent are not technical -- they are cultural, political and business-related. Getting people onto the same page can be challenging and may be an ongoing process -- information an agency can't possibly imagine releasing today may become an obvious release with no more than a change of administration or slight alteration in a regulation.

To accommodate the changing set of constraints and freedoms, any IT solution must be flexible and able to quickly adapt to new business rules and new technologies. Fortunately, IT solutions can be architected with just such flexibility.

As for the cultural, political and business needs, the data that must be reviewed and analyzed are simple to list though sometimes hard to accumulate:

  • The agency's purpose. Although agencies maintain information on many things, the information and processes that are most important to open are those related to the agency's primary purpose. Although it may be somewhat interesting to know the number of times a contractor was paid to mow an agency's lawn, it just isn't as important as information about the agency's core. In short, to provide the most value, information disclosure should align with and be related to the agency's purpose and core activities.
  • How agency information is currently collected, stored and managed. It is all well and good to mandate that an agency's expenditures be made public but if financial data is stored in multiple disparate systems, the actual technical implementation is going to be challenging. Also, agency information that is currently collected via e-mail or paper reports and is then aggregated into a spreadsheet by an executive secretary is not the best candidate for immediate release even though it may be very important data. Understanding the ways information is gathered and stored today will help determine how to make it more available tomorrow.
  • The stakeholders and interested parties. This consists of the obvious task of really finding out who the agency needs to talk to. It is an exercise in PR and marketing, not technology. Government Web sites are often built around the way the agency thinks about itself, not the way the public might think about the agency or its data. So one often finds Web sites that are structured around the agency's organization chart which is largely unknown to the public. To be effective, agencies need to find out what data their audiences and stakeholders want and how they want to access it. Unfortunately, it is not enough to just list types of stakeholders; the list needs to be prioritized. An agency may have 10 different types of stakeholders served by its Web site. To be effective the home page will need to highlight information pertinent to no more than the three or four of the most important stakeholders (the definition of "important" in this context is also a decision the agency needs to make). This isn't to say the others aren't served, just that assigning a monotone of importance to each stakeholder reduces the effectiveness of the site and misses the opportunity to better serve the most frequent or important constituents. By building around the prioritized audience types, the agency can deliver a bigger bang for the buck while still accommodating the

  • needs of all stakeholders.
  • The next step is to identify which agency data will be needed by which stakeholders and in which format.

With this groundwork in place, it is only then possible to plan the technical implementation.

Two Architectures

There are two general approaches for agencies to use when making data available, each with its own set of advantages and disadvantages, though it appears a blended approach will probably prove to be the best in the long run:

  • Creating Web sites that answer stakeholder questions and present agency data. This includes creating interfaces that end-users access to see specific reports and even generalized interfaces that allow citizens to craft their own queries against agency data.
  • Creating data feeds that give the public access to raw agency data which they can then use to generate their own reports and summaries.

A significant advantage of the first is that it allows almost anyone to get some information with little technical training or experience. But, this approach has two key disadvantages. First, to make changes or updates usually requires expensive programming resources and such efforts take time. This means the site will always tend to run behind the current need. Second, some individual or group will almost always have a new or different requirement which they cannot fulfill using the prepackaged interface. This can tie up agency resources dealing with frustration or complaints.

The advantage of the second approach is that by making the full set of data available to the public the agency leverages the time and skill of citizens who can put in the time to analyze the data the way they find important. This can maximize government IT budgets because much of the actual development is offloaded to private industry. The disadvantage, of course, is that only technical people would have immediate access to the data; the non-technical user will not be able to carry out the potentially routine simple queries he or she would like to do without enlisting the help of a technologist.

One Example

A striking example of the advantages of the second approach is Washington, D.C.'s "Apps for Democracy." Under this program, the city put up a relatively small investment to organize and administer a contest which resulted in more than 40 applications submitted -- and donated -- to the city from public and private sources. All the applications used feeds of raw data freely available from the city's data portal.

Current indications are that a combination of the two approaches might serve the widest audience: established or "canned" reports for the most commonly used data, and data feeds to access complete agency datasets to enable more complex and/or specialized manipulations. Where agencies need to choose one or the other for concentrated effort, recent suggestions are to concentrate on putting the infrastructure and data feeds in place. In a paper, Government Data and the Invisible Hand, published in the Yale Journal of Law & Technology, Vol. 11, 2008, authors Robinson, Yu, Zeller and Felten argue that federal agencies should concentrate on building the infrastructure to support data feeds rather than trying to meet every users' needs through a canned interface. The paper suggests:

  • If the next presidential administration really wants to embrace the potential of Internet-enabled government transparency, it should follow a counter-intuitive but ultimately compelling strategy: reduce the federal role in presenting important government information to citizens. Today, government bodies consider their own Web sites to be a higher priority than technical infrastructures that open up their

    • data for others to use. We argue that this understanding is a mistake. It would be preferable for government to understand providing reusable data, rather than providing Web sites, as the core of its online publishing responsibility.
    • Further, by requiring agency IT shops to use the same data feeds as are used by the public, federal IT shops would be incented to maintain that data-provision infrastructure. This approach maximizes return on investment by encouraging the development of a single, unified infrastructure responsible for managing security controls, privacy rules and trade secret protections.

    Obviously, there are also political advantages associated with the second approach. When the government doesn't take the second approach and instead gets into the business of building information-providing Web sites, it is liable to become involved in a never-ending game of catch-up in order to respond to citizen requests for changes. In contrast, if agency concentration is on providing raw data, the response to a constituency demanding a new or specific view of existing data becomes, a (polite and political) suggestion that they access the raw data and create the view themselves.

    In this way, potential IT critics can be turned into willing collaborators. At the end of the day, they may still object to the policies or actions of the agency but at least they have arrived at those conclusions in an atmosphere of greater collegial cooperation rather than what often seems like a battle royal just to extract simple data from a seemingly unwilling source.

    The Dialog

    Modern Web technology has moved well beyond being a simple way to display data. Today it encompasses myriad methods for engaging in dialogs or small or large distributed conversations. Citizens are accustomed to working with such things as WIKIs, forums, blogs and video-sharing sites and so expect them. But this is not a complete list as it is likely we will see new kinds of applications within the next five years that will be similarly transformative.

    All of these applications depend on a back-end infrastructure to store, manage and display the data so by concentrating on building an infrastructure that enables data sharing and exchange, agencies are to a certain degree future-proofing their offerings because as new forms of presentation or visualization become available, they will be built on the in-place infrastructure.


    This article has only scratched the surface of what is involved in transparency initiatives. It doesn't even directly address what might be the expected returns on investment though obviously this would be important to understand.

    Nor does it deal with the deeper internal cultural issues. For an agency accustomed to working away in relative anonymity, the idea of transparency and near real-time public feedback could be shocking and certainly might be considered a distraction. And in some cases, it would be.

    But in many ways, transparency is a logical corollary to the idea that it is a good thing to involve citizens in the business of government. And in a democracy, we understand that good government depends on citizen participation.

    David Aden
    David Aden is a writer from Washington, D.C.