When I was in school, history was about kings and queens, and the common folk were useful for a Bruegel illustration, nothing more. The focus was on those who had power and how they used it, not on those subjected to power. That focus has changed. Now some high schools use A People's History of the United States, which studies the common folk, and is considered a legitimate way to study our nation's past.
Interestingly, the State Department is undergoing a similar transformation. Thirty years ago the department's focus was on world leaders: Iran's Shah, the Soviet Union's Leonid Brezhnev, East Germany's Erich Honecker. While the department tried to follow the dissidents in those countries -- and it wasn't easy -- the attention went to those who led. Times have changed. The U.S. government has found ways to find out what is happening inside countries with repressive regimes. New communications technologies -- SMS, Twitter, the Internet -- have made the job of following the common folk far easier. And so no surprise, the U.S. government is helping dissident groups across the globe get access to new types of communications technologies. The plain folk are speaking, and the U.S. government is listening. Maybe this time the CIA can predict the collapse of the Shah's regime or the demise of the Soviet Union before it happens -- and thus help the U.S. act accordingly.
But being able to listen in on the common folk doesn't mean that figuring out what is actually going on will be all that simple. It's not necessarily easy to understand complicated situations even when you have access to all possible information (look at how the U.S. government missed the signs of the growing home mortgage crisis). Understanding complex situations is really hard if you don't fully understand the culture, or if the people are so badly silenced -- think Myanmar -- that they have no tradition of speaking up in public (or over Twitter, SMS, or Facebook), or when the government sends out false messaging in an attempt to control the messages coming from private citizens (as has happened in Tehran). In short, the public discourse in Iran or Russia on Twitter, SMS, Facebook, and other apps is likely to have lots of noise.
This makes for a really interesting problem. In the early days of the Internet, search algorithms gave back relatively poor answers: Search on "cougar" and you'd get pages that had loads of the word "cougar" written on it, but little information of interest. Or you might be sent to a website for the Ford automobile when what you wanted was information on the animal. In short, 1990s searches had little in common with the sophisticated results returned today.
The same situation is likely to occur with the State Department effort. There will be loads of chaff. Finding the germs (gems?) amongst all the wheat won't be easy, and for a time there may well be a full employment act for technically sophisticated sociologists and socially smart technologists. But just like Google upped the search ante with its PageRank algorithm, which uses "authoritative" links as the basic way to value websites, so there will be a way to separate propaganda from real tweets, and genuine comments from the government-generated ones. Tweeters might be ranked by the number of people following them, but the issue of authority matters here. While I might trust Lance Armstrong's tweets on the Tour de France, his tweets on the Haitian earthquake would much be less valuable to me than ones from Paul Farmer. I think of the National Science Foundation and the National Institutes of Health leading the nation's scientific research agenda, but it looks like at Secretary Clinton's urging, the U.S. State Department is also stimulating such research.
This is very cool. And very important. Finding out the issues that drive the citizenry of China, Pakistan, and Gaza matters to the Chinese, the Pakistanis, the Gazans -- and to the U.S. Through technology and the State Department, we are developing a handle to figure these things out.