BOSTON -- Gallup, the company that has faced intense criticism after its polling consistently understated President Barack Obama's support during the 2012 campaign, shared details of its ongoing review of its election polling methods on Saturday.
Over a four-week period before the November 2012 election, Gallup's daily tracking poll showed Republican opponent Mitt Romney leading by margins ranging from 1 to 7 percentage points, including a 4-point Romney lead just 10 days before the election. Obama defeated Romney by a 51 to 47 percent margin.
Shortly after the election Gallup's editor-in-chief Frank Newport pledged to conduct an internal review of Gallup's telephone survey methodology. As promised, Newport reviewed on Saturday the kinds of studies conducted on issues including drawing samples, interviewing voters, and how to weight data and select the likely electorate. The company pledges to make the findings of its ongoing review publicly available.
Although much analysis is now complete and set to be unveiled June 4, the investigation awaits "a major experiment" in conjunction with the gubernatorial campaigns in the fall.
"We take it seriously" when polls misfire, Newport explained on Saturday. "We've been doing presidential polling since 1936 which is what put George Gallup on the map ...The results [in 2012] certainly were not what we wanted them to be from Gallup's perspective."
Newport's remarks came during a private, on-the-record briefing in Boston attended by The Huffington Post, and a handful of other pollsters and academic researchers at the annual American Association for Public Opinion Research (AAPOR) conference.
Michael Traugott, a University of Michigan political scientist and survey methodologist retained by the company for the review, said he recommended broadening the scope and timing of the inquiry.
"Early on," Traugott said, "Frank and I had a conversation, about trying to learn what we could from an investigation of the data" collected during the 2012 campaign, "but then using that information to design a series of experiments going forward."
Traugott also said he recommended reviewing all of Gallup's methods and assessing the data collected before the 2012 campaign. "We have been assembling data on presidential approval back to 2002 as a benchmark to look at the Gallup data in relations to other data and also party identification data going back."
His analysis confirms, for example, that by "a small but consistent amount," ratings of President George W. Bush "were a little bit higher than the average" of other pollsters "and the Obama ratings were a little bit lower than the average." Those findings are consistent with a Huffington Post investigation of Gallup's results published in June 2012.
Traugott also clarified that Gallup "has agreed to make all of this information publicly available." He stressed that both he and Newport, as former presidents of AAPOR "are firmly committed to transparency. And this is an ongoing part of our relationship as well."
When pressed on whether that included releasing raw respondent-level data, Newport said that although they are "still making the final decisions, we think we'll try to make the data available as well." Such data will allow other scholars and researchers to attempt to replicate Traugott's findings and test other theories. On Saturday, Newport also asked assembled AAPOR researchers for further input on a list of 20 specific parts -- virtually every aspect -- of Gallup's methodology included in the review.
While Newport declined to reveal specific findings from the review ahead of the June 4 announcement, he shared some new information.
One of the experiments, for example, checked on potential bias associated with the Gallup brand name by running a full "shadow" sample to a recent Gallup poll in which respondents were told the polster was "Selection Research Incorporated," rather than Gallup. "The Gallup name significantly increased the response rate," Newport said, but without producing differences in the demographics of the resulting sample.
Newport also explained that earlier this year, the company changed how it asks respondents to describe their race, replacing yes/no questions critiqued in last year's Huffington Post investigation with a question in which respondents could select more than one in a list of potential answers. That change allowed for a related modification to Gallup's weighting procedure. "We think that's already had some impact on our data," Newport said.
It is difficult, Traugott explained, to reconstruct Gallup's problems "retrospectively" using the data collected last fall. Instead, he convinced Gallup to conduct experiments to determine if alternative methods would have produced different and more accurate results.
Because such experiments only make sense "in the context of a campaign," Traugott explained, they decided to undertake them during the upcoming gubernatorial campaigns in Virginia and possibly New Jersey. He stressed that the experiments would only be used for further analysis, and would not be released publicly before the election.
Three academic survey methodologists are joining Traugott in the investigation: Chris Wlezien from Temple University, an expert in likely voter models; James Wagner from the University of Michigan, an authority in telephone survey operations; and Frauke Kreuter from the University of Maryland, an expert on sampling and weighting.
Shortly after the election, Newport speculated that "it is likely that we could see significantly fewer polls conducted in the 2016 election." Asked Saturday if he was signaling Gallup's intention to withdraw from polling in the next presidential race, Newport replied, "Well, check back with us in 2016. I don't know what any polling organization is doing in 2016," given the increasing challenges to traditional pollster methods.
However, despite the months of criticism, Newport was upbeat about the potential for improvements with advances in technology. He spoke favorably about a future in which "polling is augmented by non-probability data, administrative data, social media data."
"I think it is an exciting time," he said later. "Rather than doing things the same way in any business, it's always exciting when technology in particular opens up all these other opportunities."
Before You Go
