What is the most important lesson to be learned from the emerging Daily Kos - Research 2000 polling scandal? Two prominent pollsters, Gallup Poll Editor-in-Chief Frank Newport and ABC News Polling Director Gary Langer, both chimed in last week with a similar conclusion: More disclosure is good, but poll sponsors need to do a better job checking and verifying what they publish.
First up was Newport, who is also serving this year as president of the American Association for Public Opinion Research:
I would emphasize the ultimate responsibility which rests with the entity commissioning or releasing poll data, just as a newspaper or broadcast outlet has the ultimate responsibility for what it releases or publishes. The current controversy revolves around a client-contractor relationship between Daily Kos and Research 2000. It is unclear what procedures Daily Kos may or may not have used to verify and check the data it received from the survey research firm it employed (Research 2000) before publishing it. (Daily Kos ultimately, it says, fired the research firm). Nevertheless, in general, a news or web outlet has an obligation to check and verify what it puts out. This is often easier said than done, of course. A number of publications have been burned in recent times when outside contractors or freelance writers have not followed standard journalistic procedures.
Langer, as always, was a bit more direct:
Disclosure, then, as necessary as it is, does not in and of itself assure data quality. That takes another step: The need for those who fund and then promote or disseminate these data first to dig deeply into the bona fides of the product.
Indeed to my mind the delivery of methodological details, including original datasets, should be an initial and ongoing requirement of any polling provider, not a demand only when controversy arises. That points to a more basic lesson of this story: the principle of caveat emptor.
Polling is a complex undertaking that can be produced in many ways - some highly valid and reliable, some less so, some not in the least. Anyone buying it needs to take the trouble to ascertain precisely how it's being carried out - in sampling, questionnaire design, respondent selection, interviewing, quality control, weighting and more - and to assess the appropriateness of these methods for the intended use of the research.
Just so you don't miss the point, Langer published his comments under the headline, "Running With Scissors." Snark aside, I can't quibble with the larger argument. The first and most important step in assuring data quality and integrity rests with the organizations that sponsor it. I suspect that Markos Moulitsas agrees, despite Langer's implied argument that only the "adults" of the media world can be trusted to publish surveys.
Markos has taken his share of lumps in this controversy, but we should also give him credit for taking a stand in a way that will ultimately force every ugly detail into the public domain.** Important lessons have and will be learned that have nothing to do with politics and everything to do with how to be better data consumers, whether we are paying to conduct the polls or just reading about them.
I am glad that both Newport and Langer also underscored their longstanding commitments to better disclosure, including AAPOR's emerging Transparency Initiative. Better disclosure is a critical tool for the rest of us who do not fund polling. New media brands are emerging even faster than new polling technologies, and very few of those organizations have in-house experts who can access things like sampling, questionnaire design, respondent selection, and the rest.
Those of us who are part of "The Crowd" can do our part to help others make sense of polling methodology, but only if the designs are transparent and the underlying data available.
**A semi-related post-script: In my post on Saturday I noted the "the apparent lack of a written contract" between Daily Kos and Research 2000, based on the statement in the Daily Kos complaint that they entered into an agreement "reached orally" to conduct national polls in 2009. Markos Moulitsas subsequently emailed to say that while there was no formal "boilerplate" contract, "we hashed out our agreement via email." To be clear, a legally binding contract between two parties does not require a written document.