For years the "open data" movement has promised a Field of Dreams: build it and they will come. Open your institution's databases and there will be no limit to the innovation unleashed as software developers transform this information into world-changing new applications.
Fifteen years into significant efforts by the U.S. government, the World Bank, the W3C, and many others, the achievements have been notable. Improved data transparency has proved a major boon for investigative reporters and academic scientists. Data.gov, Washington's centralized data clearinghouse, now includes more than 250,000 unique datasets.
Still, the promise of smarter decisions through open data remains largely unrealized. Relatively few software developers have worked to transform open data resources into actionable tools that create real public good. Why, when the app industry is booming with new demand and innovation, are open data providers seeing much lower uptake than anticipated?
It's largely because successful strategies for empowering software developers to actually use open data are different from the traditional definition of providing open data that organizations are familiar with. The costs of harnessing data often simply outweigh the benefits for many developers. Changing the value proposition for app developers requires a new set of best practices, one different from the previous attempts that have targeted journalists, wonks, and arcane technical experts.
At Brighter Planet, our work as a platform provider connecting data publishers with software developers to enable new sustainability applications has given us perspective on the disconnects preventing more widespread use of open data by the developer community. Beyond basic open data, there are three big things data providers can do to help grow innovation in the apps ecosystem:
- Modernize data publication. The legacy of print publications still drives the digital data release practices at many agencies and organizations. New data are published monthly or annually, in distinct installments rather than grouped as larger datasets, and when errors are found corrections are published as separate addendums. This is a nightmare for developers, who would much rather see data on a given topic combined into a single master table at a permanent URL, updated with new or corrected data as soon as it's available. Unless data are served so that machines can easily access the latest information without a human babysitter, using them in an app simply won't be worth the ongoing time commitment for many developers.
Open data projects like Data.gov do represent an important advance in the mindset of major data providers, but they can actually be misguided if the provider's goal is in fact to see their data leveraged by the growing apps movement. Clearinghouses like this were developed to improve data discoverability, but most developers will tell you discoverability has never been the major obstacle -- Google searches generally do a great job unearthing data sources without centralized directories.
Distancing data from its sources works against the goals of an apps movement that depends on immediacy, simplicity, personality, and engagement. A new vision for open data providers could see organizations like Data.gov in a more advisory role helping other agencies implement best practices for open data and developer engagement. With careful work the open data movement can and will translate into innovation in the apps space that unlocks the potential of data to inform the decisions that are increasingly migrating to mobile and web platforms. But it won't come without an intentional strategic redirection.