The big story in the 2014 Senate elections was Republicans’ success. In hindsight, the evidence was clear that Republicans were gaining support in the summer and fall, but some poll watchers were cautious when interpreting the apparent Republican gains, because of the polling community's traditional late-summer shift from polls of registered voters to polls of likely voters. An analysis testing modifications to HuffPollster’s Senate model shows that this caution was unnecessary: The Republican gains, as we now know, were real.
Polling experts know that polls of likely voters are more favorable to Republicans than polls of registered voters. The registered voters least likely to vote are often low-income voters, voters of color, urban dwellers or some combination thereof -- all groups that tend to report Democratic preferences in polls, but that often indicate they are less likely to actually vote on Election Day. (Many such voters rely on public transportation to make it to polling places, or have difficulty taking time out of the workday, which may explain why so many of them report uncertainty about whether they will vote.)
Many pollsters will report results based on registered voters until August or September, then shift to reporting results based only on likely voters as the election draws near. This change poses a problem for poll aggregators and forecasters. We want to show real trends, not trends that only show up because of a change in polling methods.
The HuffPollster model did not include a specific adjustment for registered voter polls in this year's midterms, but other forecasters and aggregators took different approaches. The New York Times' model "tweaked” registered voter polls. The FiveThirtyEight model included an explicit adjustment for registered voter polls, essentially adding to the Republican estimate (and subtracting from the Democratic estimate) for any poll that reported registered voters. A key part of the argument for making that adjustment, wrote forecasting virtuoso Nate Silver, was that “a model that’s ambivalent about the distinction might misinterpret the switch between registered and likely voter numbers as reflecting 'movement' toward the Republican Party.”
The HuffPollster model analysis shows that while there is a consistent bias against Republicans when registered voter polls are included in the model, there is not any effect on the overall trend -- that is, the change in vote share over time.
This chart shows the Republican poll estimates averaged across competitive states (including states that were once competitive but not by the end of the campaign, and including Virginia, which turned out to be competitive) for two different model runs: the final model we published, which included registered voter polls with no adjustment, and the model run with all registered voter polls excluded.
The trend lines are strikingly similar. In fact, they are nearly identical. Republicans gained in the model estimates from the end of August all the way until the election, and the gains were the same regardless of whether registered voter polls were included.
The chart shows a clear Republican advantage when the registered voter polls are excluded, but even at its widest point between July and Election Day, the gap between the two models is only 0.7 percent. Among noncompetitive races, the average trend lines have the same pattern, with the largest difference between the two models only 0.6 percent.
With few exceptions, these patterns are identical in individual states as well. Virginia’s very close election was probably the most surprising Senate result on Election Day, but the bias against the Republican candidate in the registered voter polls did not play a substantial part in underestimating Republican support. There's a notable difference between the two models, as you can see on the graph below, with an average difference of 2 percent and a maximum difference of 3.2 percent. But the gains from excluding registered voter polls still greatly underestimated the actual Republican vote.
The wide difference in Virginia, compared to the average difference of 0.7 percent among all competitive states, demonstrates that the registered voter poll bias is highly variable across states. Many states showed only minuscule average differences, particularly Arkansas (0.4 percent) and Georgia (0.2 percent). A correction to registered voter polls would have reduced bias in Virginia, but it would have created a new source of bias in Arkansas and Georgia. In this cycle, with a lower-than-usual average bias for registered voter polls, a correction would have created more bias than it resolved in our model.
Of course, each election cycle is different, and the relatively small impact on the model from registered voter polls could be unique to this year. What it does tell us is that we cannot assume that trends from the past will hold when we’re aggregating polls and creating forecasts. The good news for HuffPollster is that the registered voter issue made no substantial difference in our 2014 Senate forecasts.