Despite repeated calls to ignore pre-election polls, pollsters keep generating them and the media keeps reporting on them. Much to the pollsters’ chagrin, media outlets have used polls to determine which Republican candidates will participate in prime-time debates. Republican candidate Donald Trump has even joined the fray, unabashedly praising polls that look good for him and denigrating those that say his numbers are down.
The media and polling have a symbiotic but fraught relationship: The media needs polls to help generate content in pre-election news cycles, but then bashes the polling industry if the numbers don’t correctly predict electoral outcomes. Public pollsters need media coverage to promote their work, but at the same time blame the media for harming their industry by not adequately understanding or reporting on polls.
There is a middle ground. Some in the media try to promote smart poll consumption. We at HuffPost Pollster, along with places like FiveThirtyEight and the New York Times Upshot, discourage freaking out over single polls and encourage understanding the differences between poll methods and why that matters. We also try to be realistic about the challenges pollsters face.
The polling industry has considerable problems now, but those issues don’t render the whole enterprise useless or even remotely threatening to democracy. It’s true that no single way of contacting people covers the entire American population -- the closest you can get is to call both landline and mobile telephones, and this year the Federal Communications Commission made that more expensive and more difficult than it already was. Getting people to respond to polls is more difficult than ever, but we also have more complex methods of conducting polls than ever.
There are more polls in the 2016 races than in previous cycles. The polling industry certainly isn’t acting like an industry in decline. Supply and demand are both very high.
Part of that increase in polls is because it’s easier than ever to produce something one could call a “poll” with little cost or knowledge of scientific survey practices, passing as credible pollsters despite “cut-rate methods and carelessness in reporting critical characteristics,” according to Arthur Lupia, a University of Michigan political science professor.
At HuffPost Pollster, we aim to be inclusive. We want to include every poll that makes an honest attempt to measure opinion in the relevant population, and we don’t discount polls solely because of the methods they use. As Pollster.com founder Mark Blumenthal wrote in 2006, “we hope to provide readers with the ability to compare and contrast the results” from different methods. This is why we have the “Create Your Own” feature on all of our charts that allows you to look at the data -- or exclude data -- for specific pollsters, methods, populations and partisan affiliations.
At the same time, we agree that many pollsters are too lax in providing important details to help poll consumers assess their work. In 2010, Blumenthal wrote that Pollster would only include polls in its charts if their publicly accessible reports meet the National Council on Public Polls' “level 1” disclosure requirements, which means they should include 11 very basic elements of a poll on their releases or a public website.
For the last few years, HuffPost Pollster has been taking a broad view of the “public” part of that requirement -- we have accepted information via email, even when it’s not part of the pollsters’ public materials. However, we are now returning to the requirement that all of the NCPP’s minimal disclosure items must be available publicly before we will include that pollster’s polls in our charts.
Implementing the public disclosure requirement will take some time. We will not make any changes to our procedures for the 2016 Republican and Democratic primary races at the national or state level. We’ll continue to accept the pollsters we’ve been accepting for primary horserace questions since we’re already well into those campaigns.
For the national 2016 general election, starting Jan. 1 we will only include polls from pollsters who have all of the NCPP level 1 disclosure information publicly available either on their poll releases or on their websites. Pollsters who are members of the American Association for Public Opinion Research's Transparency Initiative will be accepted without question.
State-level presidential, gubernatorial and Senate polls will be handled a bit differently. Since there are fewer public pollsters in the states than there are at the national level, this requirement could mean that we wouldn't have enough data to produce charts for some state-level races. For that reason, at our discretion we’ll include polls that don’t fully meet the public disclosure requirement. As we’ve always done, though, we will not include polls from pollsters unless they provide us the information via email. We’ll strongly encourage the pollsters to make that information public, and we’ll make the public requirement firm at the state level in the future.
It’s true that requiring public disclosure won’t necessarily prevent weak polls from getting into our charts -- although as Blumenthal wrote, “there is probably some correlation between a pollster’s ability to answer basic questions about their methods and the quality of their work.”
Pollsters don’t agree on what makes a quality poll, but they do agree that transparency and disclosure are necessary to be considered a credible poll. So we’re reinstating the public disclosure requirement.
Questions? Reach Natalie at natalie.jackson@huffingtonpost.com.