Mississippi, Alabama Polls Missed Rick Santorum's Support

Why Mississippi And Alabama Polls Missed Santorum Support

WASHINGTON -- While Rick Santorum won the headlines with his victories in the Alabama and Mississippi primaries, pollsters were among the night's losers. Pre-election surveys in both states consistently understated Santorum's support. The reasons for that error are hard to pinpoint, but they suggest caution in interpreting future polls in Republican primaries that typically draw large numbers of evangelical voters.

Polls in Alabama and Mississippi had generally forecast a close, three-way race among Santorum, Mitt Romney and Newt Gingrich. In that respect, the data provided accurate guidance on the ballot outcomes. In Mississippi, roughly two percentage points separated the final votes for Santorum (32.8 percent), Gingrich (31.2 percent) and Romney (30.6 percent). In Alabama, Santorum's victory (34.5 percent) over Gingrich (29.3 percent) and Romney (29.0 percent) was slightly larger, but still reasonably close.

Where the surveys were consistently off, however, were in their forecast of Santorum's support. Of the eight final polls listed in the table below, none showed Santorum ahead, and six showed him in running third or lower. Moreover, in every case the understatement of Santorum's vote was beyond the margins of error reported for the poll.

2012-03-15-Blumenthal-msalpolltable.png

The size of the undecided category varied widely across the final pre-election polls, so in order to be fair, the table above also calculates Santorum's vote as a percentage of the top four Republican presidential candidates, factoring out the undecided voters. Using that measurement, the Santorum understatement ranges from 5 to 10 percentage points across the various polls. Averaged across all polls, it is about the same in Alabama (7.3 percent) and Mississippi (6.8 percent). (Omitting undecided respondents does slightly increase the margin of error, but not enough to explain away the consistent and significant pattern of survey error.)

While all of the final polls ranked Santorum too low, the understatement was smaller for the two automated pollsters, Public Policy Polling (PPP) and Rasmussen Reports. Their surveys forecast Santorum's support 5 to 6 points under his actual vote, compared to errors of 9 to 10 points for the pollsters that used live interviewers.

What explains the error, and why would automated polls do better?

One possibility is that the voters who said they were undecided broke decisively in Santorum's direction over the last few days of the campaign, but the data do not provide much support for that theory, especially in Alabama. According to network exit polls in Alabama, Santorum did just as well among those who made up their minds in the last few days as among those who decided earlier: 32 percent in each category favored Santorum. In Mississippi, Santorum ran eight points better among late deciders (40 vs. 32 percent), but still ran even with second-place finisher Gingrich among those who said they decided sooner. The three Mississippi pre-election polls had found Santorum running an average of 5 points behind Gingrich.

A second and more popular theory is that some Santorum voters were "lying" or otherwise reluctant to admit to a live interviewer that they planned to vote for Santorum. Prior research shows survey respondents tend to underreport "sensitive" behaviors to a live interviewer (such as smoking, drinking or sexual activities) that might create "social discomfort" in the conversation. So the theory goes, admitting to support of Santorum might involve a similar sense of embarrassment. Note, however, that the automated polls also understated Santorum's support in these two states, just not as much. Moreover, it is unclear why such an embarrassment factor would be significantly greater in Alabama and Mississippi than in other states that have already voted.

A third and more compelling argument is that the polls tended to sample likely Republican voters too broadly, thus under-sampling constituencies like evangelical Christians or strong conservatives who are more supportive of Santorum. Comparisons are difficult given spotty disclosure about the pre-election polls, but on this point the exit polls provide some support. For example, PPP found that 68 percent of its sample in Alabama and 70 percent in Mississippi self-identified as evangelical Christian. In the exit polls, 80 percent in Alabama and 83 percent in Mississippi identified as born-again or evangelical Christian (although note that the wording of the question was not identical -- the PPP question did not include the descriptor "born-again").

As in earlier primaries, Santorum did far better among evangelical Christians than among all other voters -- 36 vs. 26 percent in Alabama and 36 vs. 24 percent in Mississippi.

Why might polls understate the views of evangelicals or hard-core conservatives? Many polls rely on screening questions that ask voters if they are "likely to vote" in the upcoming primary, a method that generally overstates their true intent.

A too-inclusive screen for likely voters in a GOP primary would tend to include too many independent and moderate voters who lean Republican but do not typically vote in primary elections. Automated polls may have less of a problem on this score, once again because they lessen the "social discomfort" for non-voters of hanging up the phone out of disinterest or admitting they do not plan to vote.

Of course, we have not seen as big a Santorum understatement in previous primary and caucus states. Nate Silver reported that his polling-based forecasts understated Santorum's vote in Iowa (by 5.5 points), Tennessee (4.4 points), South Carolina (3.1 points) and Ohio (2.7 points). There were also big Santorum understatements in three states where PPP was the only pollster: Colorado (12.7 points), Minnesota (11.6 points) and Missouri (8.3 points).

What these states have in common is large numbers of evangelical Christians voting in the primary or caucus: Mississippi (83 percent), Alabama (80 percent), Tennessee (76 percent), South Carolina (65 percent) and Iowa (57 percent). Entrance polls were not conducted for caucus states other than Iowa.

In fact, the correlation between the percentage of white evangelical voters and the Santorum understatement is quite high, as seen in the chart below. Generally speaking, the higher the percentage of white evangelical Christians, the more the polls have tended to understate Santorum's support.

2012-03-15-Blumenthal-errrorvsevengelical.png

Silver reviewed some additional theories for the Santorum error, but ultimately concluded that it is premature to assume a widespread problem and simply "add a couple of points to Mr. Santorum's numbers as a default." That caution seems appropriate, especially since few of the upcoming Republican primary states are likely to draw anywhere near as many evangelical Christian voters as Alabama and Mississippi did.

One upcoming primary state may be an exception. In 2008, the Louisiana exit poll found that evangelical Christians were 56 percent of Republican primary voters. So far, polls on the state's March 24 primary have been sparse, but given the past results, we might be especially cautious about interpreting the Louisiana polls still to come.

Before You Go

Popular in the Community

Close

What's Hot