"No one was ever promoted for disclosing information."

US Government Employee

Industry analyst Gartner describes web services based on open government data as having a "greater potential effect on the ability to transform government than anything else in the Web 2.0 world". In his technology platform, Barack Obama has made groundbreaking promises related to increasing government accountability by: i) publishing data in open formats; and ii) using online tools to involve citizens in government decision making. These transformative ideas have not yet spread to politics in Canada.

As citizens, we trust that money is being wisely spent on the systems that run our country. We trust that the people governing us have the skills, time, and information they need to make the best decisions. We trust that bureaucracies are well-designed and that the people in them are motivated to make those bureaucracies better. Unfortunately, it's hard to trust what you can't see. By publishing information in open, machine-readable formats, governments can take a powerful step towards building public trust. By sharing information, governments can start to channel the expertise of the citizenry outside of the civil service to build more effective and inclusive ways of running the country.

The non-profit VisibleGovernment.ca is working to make online tools for civic participation based on open government data a reality in Canada. This article describes why open government data is not only a requirement for greater government transparency, but also a valuable investment in our country's infrastructure.

Obama's Promise

The Obama campaign made international headlines for tapping into a wellspring of online contributions. The campaign raised $742M dollars in total, almost twice as much as the more traditionally-funded McCain campaign. The majority of this money came through online contributions. The Washington post identified Lynne Bailey, a 52-year-old mother of two who gave a total of $120.40 in mostly $10 increments, as a typical online donor. Like Bailey, 90% of Obama's donors were small scale donors, contributing less than $200 each. These small scale donations accounted for 57% of funds raised in Obama's presidential campaign. This wealth of donations from ordinary Americans could not have been tapped without the Internet.

The campaign also made headlines for creating online tools that allowed their army of on-the-ground volunteers to self-organize. The Obama volunteer website, called the most important video game of 2008 (http://www.techpresident.com/blog/entry/33178/mybo_the_video_game), allowed campaigners to post their activities, organize house meetings, share their experiences, and distribute responsibility for making phone calls encouraging voters to get to the polls. The tools showcased in the Obama campaign have forever changed the way politicians are elected. What may be more interesting is how the online tools for governing the country instigated by Obama's team and policies will change the way governments around the world work.

Well before he was a presidential nominee, Obama was defining strategies for using the Internet to improve government openness and accountability. In 2006, Obama, along with senators Tom Coburn, Tom Carper and John McCain filed the Federal Funding Accountability and Transparency Act which mandated the creation of a searchable website of all government spending by January 1, 2008. After his initial refusal, citing that the website would be too costly to build, President Bush signed the bill on September 26, 2006. The website, fedspending.org, was released ahead of schedule in December, 2007. Not only does the website provide an easily navigable interface, it provides an application programming interface (API) for external developers to access that information and build tools of their own. According to the US watchdog group OMBWatch (http://www.ombwatch.org/), 11 states have since created similar state-spending websites and 24 other states are working towards that goal.

Building on these themes of openness and accountability, Obama included the following in his Technology Platform under the heading Create a Transparent and Connected Democracy: "Obama will integrate citizens into the actual business of government by:

• Making government data available online in universally accessible formats to allow citizens to make use of that data to comment, derive value, and take action in their own communities. Greater access to environmental data, for example, will help citizens learn about pollution in their communities, provide information about local conditions back to government and empower people to protect themselves.
• Establishing pilot programs to open up government decision-making and involve the public in the work of agencies, not simply by soliciting opinions, but by tapping into the vast and distributed expertise of the American citizenry to help government make more informed decisions."

Making government data available online in accessible formats is a powerful idea. For this transformation to take place, however, governments have to be willing to accept feedback and analysis based on the information generated. Thus, the second point of a willingness to "tap into the vast and distributed expertise of citizenry" becomes vitally important.

Open Systems Allow External Contribution

Wikinomics, by Dan Tapscott, opens with the story of Goldcorp, an Ontario mining company which faced bankruptcy in 1999. A young mutual fund manager, Rob McEwan, had become majority owner of Goldcorp after a messy takeover battle several years earlier. While early test drilling had indicated substantial gold deposits, years of searching by Goldcorp's engineers and geologists had not found the gold's exact location. Inspired by the story of Linux, where Linus Torvalds co-ordinated the development of a world-class operating system over the Internet, McEwan published Goldcorp's geological data and announced a challenge with half a million dollars of prize money. The analysis of the results submitted by geologists from all over the world created almost 9 billion dollars of market value.

Tapscott emphasizes that opening up that data took tremendous bravery. The company had to admit that it did not know how to find its own gold. McEwan "realized that the uniquely qualified minds to make new discoveries were probably outside the boundaries of his organization, and by sharing some intellectual property he could harness the power of collective genius and capability". What if governments in Canada were to take the same attitude?

There have been several examples in the US and UK of web sites that use government data, opened up through APIs, gathered by scraping, or compiled by citizens themselves, that allow citizens to contribute analysis, expertise, or local knowledge for public benefit. A short list of these sites includes:

Maplight.org: produced by the Sunlight Foundation, this website analyzes the relationship between contributions and votes in the US congress. The site shows simple histograms, per bill, of donations by groups for and against the bill, against a histogram of votes. The site allows citizens to look for trends between contributions and votes by bill and by law-maker. By making this information visible, the site is effectively crowdsourcing the function of a contribution watchdog, enabling issue detection and discussion by both bloggers and the mainstream media.

PeerToPatent.org: the goal of this site, a project of the New York Law School, is to relieve some of the burden of over-worked officers at the US Patent and Trademark Office by tapping into an online community of civilian experts. These experts search for and explain prior art, as well as vote on the strength of patent applications. A year into the pilot, the PeerToPatent system reported over 2,000 citizen reviewers with the average reviewer spending six hours reviewing each patent.

FixMyStreet.com: this site, produced by the UK non-profit MySociety, allows citizens to report public safety and nuisance issues in their neighbourhood such as graffiti, potholes, or bad lighting. Citizens can then subscribe to an RSS feed to receive updates on a particular problem by the town council. This website not only provides citizens with an easy way to report and monitor problems, it reduces the burden on authorities who are less likely to have to handle repeated complaints submitted individually. Further, the number of subscribers to a particular problem can be used as a rough indication of public interest in an issue.

These sites are early experiments in the field, and represent the iceberg tip of what may be possible.

Open Systems Make Failure Less Costly

Finding the best ways to analyze government information and collect value from public feedback is going to take a lot of experimentation. In Here Comes Everybody, author Clay Shirky qualifies the probability of a successful solution as a scalar distribution pattern where there's a very large number of failures, some modest successes, and a few solutions that will do amazingly well. Being prepared to accept a lot of failures is the key to finding the successes.

Government bureaucracies are failure-averse for very good reasons. Public scrutiny and the spectre of being accused of wasting taxpayer funds make for a cautious environment where money is only spent on guaranteed successes. By publishing data in open, standardized formats, governments can off-load the costs and stigma of failure to external organizations. Like Goldcorp, governments can take the open approach to innovation by challenging advocacy groups, the nascent community of armchair egovernment-geeks, and the for-profit market to build a better way. The government can then take advantage of the value created by the best solutions. Solutions that don't work can die quietly, without any tax dollars having been spent.

Open Systems Create New Markets for Innovation

Unrestricted access to government data will create new markets for innovative ways of presenting, analyzing, and combining that data. Some creative companies will eventually find profitable ways of using this information to generate value. Some of these discoveries will inevitably be even more valuable as public goods.

For the last 10 years, Cisco has grown by accepting the fact that, even though it employs some of the most brilliant people in the industry, there is a very low probability of the future's most successful idea being generated in-house. Instead, Cisco's model is to buy the cream of innovation, effectively outsourcing their technology research and development to the venture capital market and startups. If the government were to get in this habit of buying the most successful innovations in displaying and using government data, and turning them into public goods, it would create a thriving market for more of the same. Further, if the government were to open source the tools that it buys, it would create an expanding base of software components for building increasingly sophisticated tools.

An Architecture for Open

The UK's Power of Information Task Force has proposed an application framework for implementing government transparency. In a thoughtful blog post this past June, Richard Allen proposed the following re-visioning of the way that the data in a government website is used. Instead of a closed model where the presentation, analysis, and data layers are locked together, Allen presents a model with access layers between data, analysis, and presentation, and an interaction layer laid over top. These access layers give third parties the flexibility to hook into the data directly to provide their own analysis or to use information from the government's analysis layer to provide their own presentation interfaces. Finally, the interaction layer allows people to discuss the information and provide feedback. Figure 1 illustrates the difference in architecture.

Figure 1: Traditional Approach vs. Power of Information Architecture

As a concrete example of how this model can be applied, Allen presents the evolution of tools around the UK's parliamentary Hansard, a record of parliamentary proceedings, as seen in Figure 2.

Figure 2: Hansard in the Old and New Model

Originally, www.parliament.uk took an integrated approach where the Hansard data was "wrapped up with Parliament's own analysis output and presented to the public in an official website." The innovation of a click-use license for copyright allowed a citizen-managed project called publicwhip.org.uk to begin scraping the data and providing it for public use. Allen describes the process: "An access layer has been created for Hansard with a screen scraper and Click-Use license to address both technical and copyright issues. The scraped data goes through an analysis process at publicwhip.org.uk. Access to the output of this analysis process is offered by means of XML data under a Creative Commons license. An API has been produced to make it very easy to get this data. TheyWorkForYou.com provides a very good and popular presentation layer for this content. The data as reworked by TheyWorkForYou is also commonly presented in many other places on the web such as MPs' personal sites. There is a comment facility built into TheyWorkForYou to provide a layer of interaction around the content. It is also cited in many blogs that generate their own interaction as well as featuring in mainstream media stimulating further discussion. The new architecture now provides a platform for more innovation around the Hansard data set with very low barriers to doing this".

David Robinson, of Princeton's Center for Information Technology Policy, takes the concept of fitting access layers into existing government IT architectures one step further. In his paper Government Data and the Invisible Hand, Robinson argues that intra-departmental reporting channels should be exposed to the public, who can provide external validation to complement internal checks and balances.

If this model were followed by the Canadian federal government, data provided to the Auditor General for fulfilling its mandate of "holding the federal government accountable for its stewardship of public funds" would be opened up to access by external agencies
Open Source Tools for Open Data

Open source licensing for the tools that present and use government information takes the concept of transparency to the level of the source code. It enables public scrutiny of the presentation and analysis methods. In the words of one of OMBWatch's recommendations for President Obama: "...agencies should have a policy to exercise a preference for open source software for government activities as a means to improve stability, transparency, metadata quality, and cost-efficiency. Open formats for government information and open software applications will enable collaboration between agencies and will increase civilian oversight, participation, and use of taxpayer-funded resources".

South Africa, Brazil, and China have begun to adopt policies favouring open source. According to a 2007 report in the open source online magazine Tectonic, "all new software developed for or by the [South African] government will be based on open standards and government will itself migrate current software to FOSS. This strategy will, among other things, lower administration costs and enhance local IT skills".

The call for open, standardized APIs for government data creates an incentive for governments to provide new systems for publishing that data. Many government bodies will be facing the same problems around producing reliable data streams complete with sensible meta-data.

In the Canadian federal government, departments are typically left to come up with their own solutions, paid for out of each departmental budget. This leads to a proliferation of redundant systems built according to the preferences of each department's information technology consultant. Were departments to pool their resources into a joint open source effort to create data publishing systems, it would save costs and create a foundation of inter-departmental co-operation. The software produced would also benefit from the public scrutiny, oversight and contribution of Canadian citizens. Further, it would be free for governments around the world to adopt, creating the potential for contribution from citizens world-wide.

Open System Roadblocks

The rewards of a civil service career are asymmetrical and civil servants often feel that they live in a fish bowl. This fish bowl is made of a particular type of filtered glass: one where only the bad light gets through. Overwhelmingly, the disclosed information that gets publicized by the media is the negative, career-destroying kind. Information that points to success and improvement are rarely publicly celebrated. This is something that has to change.

Recognizing that the incentives against transparency outweigh the incentives for, OMBWatch has recommendations for institutionalizing open. These include:

• having the government leader instruct agencies to request sufficient resources in funding, personnel, and technical capacity, to implement the vision of a more transparent government
• making transparency part of federal job evaluations where it is part of the job description
• implement directives protecting whistle-blowers who disclose waste, fraud, or abuse within an agency
• creating a system of transparency scorecards for rating agencies
• giving out transparency awards to celebrate achievements and best practices

Beyond these recommendations, external bodies that use government information should, as much as possible, build systems that create heroes rather than scapegoats. Individuals who find ways to save money, increase efficiency, or deliver a valuable service in an innovative way should be publicly rewarded, either through external financial compensation or public recognition.

Public service was, at one time, thought of as a calling. If civil servants who improve the way government functions are celebrated with the same media reverence granted successful businessmen, perhaps it may become one again.

Promoting Citizen Services Based on Open Government Data

The non-profit VisibleGovernment.ca was officially incorporated in December, 2008 to promote online tools for government transparency. One of its founding principles is that while there is a sound case for open government data, a third party organization is needed to raise awareness of the issues and to marshal public support.

VisibleGovernment.ca's strategy is to build a limited number of pilot projects to gain visibility in Canada for the power of open government data to transform the relationship between citizen and government. The I Believe in Open pilot challenged candidates in the 2008 federal election to pledge to five aspects of government transparency. The site also collected signups from voters, organized into ridings, so that member of Parliament (MP) candidates could see the support level in their area. 400 MP candidates signed our online pledge, 38 of whom were elected.

Another VisibleGovernment.ca pilot is a tool for visualizing federal government travel and hospitality expenses. The project gathers data from tables published in a variety of formats spread over 100 different department websites, and creates an interface that citizens can use to visualize this information, compare departments, and see trends over time. The project provides an RSS feed so that other groups can use the data to create tools of their own. With help from volunteers in the Montreal high tech community, over 30,000 records have been collected so far. A Montreal web development company specializing in data visualization has volunteered to do the visualization website, which we intend to launch at the Social Media for Government conference in Ottawa this February.

The long term goal of VisibleGovernment.ca is to direct money and attention to external projects that further our mission via grants and contests. By being a catalyst driving public support of open government data, and pushing the envelope of innovation for ways of analyzing and presenting that data, we hope to create a network of active citizens who believe that open should be the normal state of governments.

Ways Forward

In the last two weeks, more than one grass-roots forum has appeared to advance a new era in civic participation. Toronto's ChangeCamp and Montreal's Forum Ouverte are two such movements calling for an open exchange of ideas around using technology to re-define the role of the citizen. The spontaneous emergence of these groups shows the demand for new ideas and tools in the Canadian government.

VisibleGovenrment.ca seeks the expertise and participation of grass-roots groups, advocacy organizations, and citizens across the country to make online tools for civic participation based on open government data a reality. If you share this goal, here are some concrete steps for action:

• host a ChangeCamp or Forum Ouverte in your city
• research resources and strategies on the VisibleGovernment.ca, Sunlight Foundation, or MySociety websites
• contribute to a VisibleGovernment.ca project
• start a dialogue with your public representatives on how they can be more open

For more information, visit the Visible Government.ca website, join our online discussion group, or email the author.