Another long day for PPP, as the New Republic's Nate Cohn reports about previously undisclosed aspects of their methods. Opposition to airstrikes in Syria grows in the US (and is widespread in Syria. And Lee Miringoff steps up to defend public pollsters. This is HuffPollster for Thursday, September 12, 2013.
PPP’S ‘UNSCIENTIFIC AND UNSETTLING’ METHODOLOGY - Nate Cohn: "After examining PPP’s polls from 2012 and conducting a lengthy exchange with PPP’s director, I've found that PPP withheld controversial elements of its methodology, to the extent it even has one, and treated its data inconsistently. The racial composition of PPP’s surveys was informed by whether respondents voted for Obama or John McCain in 2008, even though it wasn’t stated in its methodology. PPP then deleted the question from detailed releases to avoid criticism. Throughout its seemingly successful run, PPP used amateurish weighting techniques that distorted its samples—embracing a unique, ad hoc philosophy that, time and time again, seemed to save PPP from producing outlying results. The end result is unscientific and unsettling."
Lack of consistency - "Of course, it's impossible to prove that PPP weights toward a desired result. But what’s so troubling is that it’s totally possible. No other pollster employs a truly ad hoc approach, with the flexibility to weight to whatever electorate it chooses, while allowing the composition of the electorate to fluctuate based on the inconsistent and subjective application of controversial or undisclosed metrics. In PPP's own words, they “don’t have any specific rules” about how to weight from poll to poll. And within that framework, it's a little absurd that PPP reserves the power to wield a weighting bludgeon, like considering the last election, whenever and however the firm feels like it. And while that inconsistency prevents anyone from proving that they weight toward a desired result, it also prevents PPP from proving that it does not." [TNR]
Reactions:
PPP Director Tom Jensen, in a post containing his email conversations with Cohn: "[Cohn's] basic point is that while our results are generally accurate, he doesn't like how we get to them. The simple reality is that in an era of record low response rates, how you weight your polls is a very important part of whether they end up accurately predicting election results or not. Pollsters have different ideas about how to do that, and our methodology is unique within the industry. But it's worked for us, it has for years, and we're going to stick with it. We've never really cared about doing things the way everyone else does." [PPP]
Nate Silver: “The main problem with @ppppolls is that their approach to polling is extremely ad hoc. Ultimately, that ad-hockery stems from a lack of appreciation/understanding for the statistical fundamentals behind polling. @ppppolls proudly endorse the idea of using of ‘gut feeling’ in conducting their polls. But ‘gut feeling’ is often used to excuse all sorts of conscious and unconscious biases. Statistical methods certainly do require *judgment*. But good judgment is based on evaluating PROCESS… not RESULTS.” [@fivethirtyeight]
GOP pollster Logan Dobson: “If I set up a "political guessing society" & correctly guessed lots of election results, that'd be great, but it wouldn't be a Polling Firm” [@LoganDboson]
Political scientist Michael McDonald: “@ppppolls looks like no good deed goes unpunished. You've provided a lot of ammunition to @Nate_Cohn where other pollsters would be mum” [@ElectProject]
Democratic pollster Nick Gourevitch: “Big diff between PPP & most private Dem polling firms: Most of us don't release polls publicly just for the sake of showing we are accurate.” [@nickgourevitch]
Democratic pollster Matt McDermott: “If you have a hard time explaining methodology, you probably don't have one, or at most have a weak one. And that's the ballgame in polling.” [@mattmfm]
Daily Kos founder Markos Moulitsas: “Love seeing Democratic pollsters jump on @PPPPolls, they're the ones whose price-gouging model has been disrupted.” [@markos]
Former DCCC data/targeting director Aaron Strauss: “.@Nate_Cohn since you asked (http://t.co/XGyphr5Rgg), I think @ppppolls methodology is pretty solid. RBS w/ weighting ranges & '08 ballot.” [@aaron_strauss]
Political scientist Drew Linzer: “Sounds like PPP is using an intuitive "shrinkage" weighting, between the sample and known populatn. That's defensible….@Nate_Cohn Have you ever worked at a polling firm? When you have small samples, there's judgment involved. There has to be.” [@DrewLinzer]
Democratic pollster Anna Greenberg: "You can't bill yourself as the 'most accurate pollster' if you can't deal with the scrutiny of your methodology." [@Anna_Greenberg]
OUR TAKE - Cohn's piece boils down to two central complaints about PPP: That their procedures for weighting data are inconsistent and allow for more subjective judgements about who will vote, and that they hid a critical aspect of those procedures from public view. Let's take those in order.
Subjective judgement - To be fair, there is considerable subjectivity in preelection polling, more perhaps than most pollsters like to admit. Election pollsters generally fall into two broad categories: Some begin by sampling all adults, weighting the demographics of those samples to match Census estimates. They then select likely voters based on questions answered by the respondents, letting their demographics vary from poll to poll. Others begin by sampling from lists of all registered voters or by first screening out non-voters before conducted an interview, requiring more subjectivity when deciding how to weight the demographics of the selected sample. National media pollsters tend to use the first philosophy, private campaign pollsters the second (though there are many exceptions for each). But both have to make subjective judgements about something important, be it the level of expected turnout or the questions that best "model" that turnout (for the former) or about some method of determining the demographics of the electorate (for the second).
Extreme 'ad-hockery'- What Cohn reports, in essence, is that PPP falls at the extreme of the subjective "gut feeling" end of the pollster spectrum. PPP is not alone on that score. Nine years ago, the Washington Post reported that pollster John Zogby weighted his samples based in part on "what is happening on the ground." They also noted at the time that "most pollsters cringe at such extra-curricular adjustments." PPP's defense is that its approach has proven to be "generally accurate," which it has. The problem, however, is that at some point it gets hard to distinguish the pollster's judgements from the poll's measurement of voter preferences. [WaPost]
Hidden method - The biggest revelation in Cohn's piece is that PPP "take[s] into account" self-reports of past voting preference (for Obama or McCain on their 2012 surveys) when weighting by demographics. In other words, when they thought the partisan balance in the reports of past voting were off, the would discard interviews in order to bring past vote preference into balance. While PPP did share this information with Cohn, they never before disclosed that aspect of their methods and they intentionally omitted the past vote question from most of their statewide 2012 polls. In the email exchange released by PPP, Cohn had to ask three times for an explanation of the odd pattern he had observed before Jensen finally revealed the reliance on the past vote question (Cohn tells HuffPollster via email that one of his early emails is omitted from the "full" listing). Yes, PPP discloses a lot of detail about their data, but the trouble is, PPP hid a very important aspect of their procedure which, without Cohn's prodding, would remain unknown. Jensen is right that in the current era, "how you weight your polls is a very important part" of what makes a poll accurate, but that also means that pollsters need to fully disclose how they weight.
COHN POSTS EPILOGUE: Cohn adds, in a second post: “Jensen posted our full email exchange. I'm totally fine with this, since, well, my original piece accurately represents the exchange. Unfortunately, Jensen suggested that I hadn't included his explanations when, in fact, I did. And when asked for a specific example of something that I should have posted, Jensen refrained from offering a reply. And if PPP thinks there was some exonerating detail in our exchange, they should have just said so. But there's not one. The real reason Jensen posted the exchange: To make it look like he was really transparent. But Jensen only explained his methodology after I provided data that truly required an explanation. And there's a reason PPP didn't want to mention this until now: it's a bad practice. Selectively using the '08 election isn't too much different from weighting to a desired result; it's not too much different than weighting by party ID; and it's particularly egregious to incorporate it into race weighting, rather than weight [simultaneously].” [TNR]
OPPOSITION TO SYRIA INTERVENTION GROWS - Emily Swanson: “Support for military action against Syria continues to plummet, according to a new HuffPost/YouGov poll. And even the few Americans who back airstrikes want to avoid them if Syria is willing to give up its stock of chemical weapons. The latest poll found only 12 percent of Americans who said they supported strikes, compared to 61 percent who said they did not. That's down from 25 percent who backed airstrikes in another HuffPost/YouGov poll conducted two weeks ago….Even among the small group of respondents who said they support airstrikes, 66 percent said that the U.S. should unleash military force only if Syria does not relinquish its chemical weapons.” [HuffPost]
SYRIANS ALSO DISAPPROVE - Doug Rivers: “YouGov operates market research panels around the world and over the past week has interviewed 835 Syrians (500 mobile, 335 internet). Many say their homes have been bombed and are displaced (one unlucky person four times!)....This is not a representative sample: three-quarters are male, over half are under 30 years old, and just under 50 percent say they have a university degree. Most are poor (with monthly incomes under $266) and Sunni Muslims. But the sample also contains 248 supporters of the Assad regime, 152 opponents and a larger number who support neither side or prefer not to tell us…. More opponents of the regime strongly disapprove of a U.S. military strike than favor it. 81 percent of government supporters, as well as 56 percent of those who prefer not to say. There's little evidence that ordinary Syrians favor an attack. In fact, distrust of America is nearly unanimous among Syrian poll-takers. Only 7 percent of those interviewed thought that the U.S. government was ‘a friend of the Syrian people.’” [Huffpost]
LEE MIRINGOFF DEFENDS THE ROLE OF PUBLIC POLLING - Miringoff, using NYC mayoral polling as a case study: “Public polls serve a useful, and yes, even a vital function in today’s high tech politics. They offer, if conducted well, an insightful narrative of a campaign. They guide journalists and poll-watchers about the dynamics shaping the electorate. What are the issues driving voters? How are they reacting to campaign developments? What is the composition of the electorate and the appeal of the candidates? This primary, it was extremely interesting to see how Democratic voters were assessing term limits, stop and frisk, affordability, the 12-year incumbency of Michael Bloomberg, and the television campaign ads… the so-called ‘Dante effect’…Public polls also let the public in on the secret of what the private campaign polls are showing and provide insight about how candidates shape their strategies to survive the rough and tumble world of Big Apple electoral politics.” [Marist]
HUFFPOLLSTER VIA EMAIL! - You can receive this daily update every weekday via email! Just enter your email address in the box on the upper right corner of this page, and click "sign up." That's all there is to it (and you can unsubscribe anytime).
THURSDAY'S OUTLIERS' - Links to more news at the intersection of polling, politics and political data:
-Congressional approval rises "significantly" to 19 percent. [Gallup]
-Sean Trende looks at the results of the Colorado recall elections. [RCP]
-A Reason/Rupe poll finds opposition to raising the debt ceiling. [Reason]
-Americans don’t see the economy as any more secure than it was five years ago. [Pew Research]
-Geoff Garin, pollster for NYC mayoral candidate Bill Thompson, says his client is "being railroaded by a voting system as flawed as Florida's." [@GeoffGarin]
-Jennifer Agiesta ponders mistrust of pollsters and declining response rates. [AP]
-Pope perpetually pretty popular in Pew polling. [Pew Research]
-Republicans are reading The Victory Lab to study for the 2014 midterms. [New York magazine]
-Stuart Rothenberg takes issue with a poll of a Michigan House primary. [Roll Call]