WASHINGTON -- A new poll of Massachusetts voters that shows a surprising 9 percentage point lead for Republican Sen. Scott Brown against Democratic challenger Elizabeth Warren used an unorthodox ordering of questions that drew criticism even from Brown's pollster.
The survey conducted by the Suffolk University Political Research Center for Boston television station WHDH (7News), from Feb. 11 to Feb. 15, gives Brown a 9 percentage point lead over Warren (49 percent to 40 percent) with 2 percent opting for another candidate and 9 percent undecided.
The Suffolk numbers are very different than results from second poll released on Tuesday, conducted by the Massachusetts think tank MassINC on behalf of Boston NPR station WBUR, that gave Warren a 3 percentage point edge (46 percent to 43 percent) that was not large enough to be considered statistically significant. Two other surveys conducted in late 2011 also showed Warren leading by small, single-digit margins.
One possible explanation for the divergent results comes from the questions asked just before the vote preference question. Both the Suffolk and MassINC polls begin by asking voters whether they have favorable or unfavorable impressions of each of the candidates. The MassINC poll then immediately asks about vote preference, while the Suffolk poll also asks the following:
Q9. Does Senator Scott Brown deserve to be re-elected or is it time to give someone else a chance?Q10. What is the first word or phrase that comes to your mind when you hear the name Scott Brown?
Q11. What is the first word or phrase that comes to your mind when you hear the name Elizabeth Warren?
Q12. Does Elizabeth Warren have the experience to be a United States Senator?
Q13. Is Scott Brown a leader in the United States Senate, or a follower?
Q14. If the General Election for United States Senate were held today and the candidates were Republican Scott Brown and Democrat Elizabeth Warren for whom would you vote or towards whom would you lean at this time?
The two open-ended questions (Q10 and Q11) are not troublesome, but the order of the other items is. While each uses neutral wording, they may have the collective side-effect of reminding respondents, three times, that Scott Brown is an incumbent senator. They may also raise doubts about Elizabeth Warren's experience while planting the suggestion that Brown has been a leader in the Senate, not a "follower."
The issue on the Suffolk survey is similar to a criticism leveled earlier this month by Mitt Romney campaign pollster Neil Newhouse about a recent ABC News/Washington Post survey. Newhouse argued that just before measuring the Barack Obama-Romney sentiment, the pollsters asked a series of questions that "introduced specific negative information about Governor Romney." These included a set of questions about three candidates -- Romney, Newt Gingrich and President Obama -- as well as several more specific items about Romney. These included a question about whether, given his "work as a corporate investor ... Mitt Romney did more to create jobs or more to cut jobs," and a question asking whether Romney "is or is not paying his fair share of taxes" having "paid about a 14% federal tax rate on income of about 22 million dollars last year."
Asked to comment by the The Huffington Post, Newhouse -- who is also the pollster for Scott Brown -- said that the criticisms he leveled against the ABC/Washington Post poll would "absolutely" apply to the Suffolk poll, "though not to the same degree" as the ABC/Post poll.
David Paleologos, the director of the Suffolk University Political Research Center, told The Huffington Post that he doubts the questions he asked just before the Brown-Warren vote question "will have an order effect, per the [academic] literature," which he regards as "inconclusive." He also noted that responses to the two questions asked just before the vote "broke fairly close," adding "it's not like [these] questions broke 70-30 or something like that."
Peleologos also pointed to the accuracy of their past polling, particularly the survey released just five days before the January 2010 special election, one of the first to show Scott Brown leading Democrat Martha Coakley. Although "mindful of what some of the other polls have shown," he expressed confidence that the Senate race has been "trending to Brown" who has "been doing a lot of radio."
Survey researchers have long understood that the order of individual questions can matter as much as their wording. The most authoritative research on the subject was conducted by two academics, Howard Schuman and Stanley Presser, beginning in the 1970s. Using a series of survey experiments, they found that the order of questions alone could sometimes affect the answers provided by respondents.
In their book, "Questions & Answers in Attitude Surveys", Schuman and Presser explain that general questions appear to be more sensitive to these "question order effects" than more specific probes. That evidence inspired a simple lesson taught to generations of aspiring pollsters. To quote another widely assigned text, Survey Methodology, "when asking general and specific [attitude] questions about a topic, ask the general question first."
These findings persuaded pollsters like Newhouse to adopt a practice common among researchers hired by political campaigns: Ask candidate favorable ratings first, then questions about vote preference and then more specific items about candidate characteristics and issues.
For media pollsters, who often use their surveys to probe a wide variety of subjects, the task of ordering questions can be far more challenging and the "best practices" not as clearly defined. In the weeks leading up to an election, media pollsters will typically ask their horse-race questions at the very beginning of the interview, but at other times questions about presidential approval, the condition of the economy or other policy issues take precedence.
Mindful that the academic experiments have found effects that were neither "rare" nor "pervasive," as Schuman and Presser put it, many media pollsters have concluded that an occasionally unorthodox ordering of questions rarely does practical harm.
For example, in defending their survey, ABC News pollster Gary Langer argued that Obama's share of the vote tracked closely both with his approval rating, as measured near the beginning of the same survey, and with results obtained by other polls fielded at about the same time.
The same argument is tougher for Paleologos and the Suffolk University Poll, since they have produced a margin favoring Scott Brown that is a net 12 percentage points different than the result on the MassINC survey conducted just a few days earlier.
We will likely see more surveys soon, and those may show that Brown has rebounded from the narrow Warren advantage measured in late 2012. But for now the current standing of the candidates in the Massachusetts Senate race appears uncertain.