The Most Dangerous Poll?

Three caveats to the apparent public support for Trump's proposed Muslim ban.

For months, establishment Republicans have speculated--"prayed" may be the better term--that Donald Trump's support would prove shallow and transient. A Bloomberg Politics/Purple Strategies poll released Wednesday, on his proposal to bar Muslims from entering the U.S., seems to dispel those hopes: fully 65 percent of "likely 2016 Republican primary voters" surveyed endorsed Trump's plan.

"We asked people the question, and then we gave them more information...the numbers did not change," commented Bloomberg's John Heilemann. In fact, some 37 percent of Republicans polled went so far as to report that Trump's policies make them more likely to back his candidacy.

With so much attention to Trump's statements in recent months, it may be that his support has indeed become well-considered and durable. But with a CBS/New York Times poll released Thursday showing 64 percent of those Republican-leaning voters still undecided, and a third of them "either concerned or frightened" about the mogul, it's worth looking past the headline numbers to ask what inferences the Purple Strategies poll's methodology really supports. Three pitfalls suggest themselves.

The first is the problem, prevalent in instant polling since the start of this election cycle, of selection bias. In online polls like the one conducted by Purple Strategies, "likely Republican voters" merely self-identify as such; some may be neither likely to vote, nor even registered Republicans.

For predictive purposes, it might not matter. Trump might well attract new voters to the process from inside and outside the party, and some key early contests allow cross-party voting. However, there's a history of verified past Republican-primary voters showing less enthusiasm for Trump than casual survey respondents. Those seeking ammunition in such polls against the Republican party at large--many of whose leaders have loudly decried Trump's proposed ban--should proceed with caution.

The second concern is the propensity of the language in the Purple Strategies questionnaire toward anchoring bias. This is largely a question of how information and questions are sequenced.

The survey's inquiry into Trump's proposal begins straightforwardly:

"This week, Donald Trump called for a temporary ban on all Muslims who are citizens of foreign countries from entering the United States. Do you favor or oppose this proposal?"

Key terms here include "temporary" and "foreign countries." These words accurately characterize Trump's proposal, but strip it of the rhetorical excess that have gained it such media attention. With expectations of the survey "anchored" in this dispassionate tone, 51 percent of Republicans responded that they "strongly favor" the proposal; 14 percent chose "Not so strong favor" [sic]. The figures for Democrats were 14 percent and 4 percent, respectively.

What happens when that rhetorical excess is introduced? Respondents were next presented with two rotating paragraphs that provided a taste. The first paragraph simply conveyed Trump's own words:

"Donald Trump called for the temporary ban, saying 'Until we are able to determine and understand this problem and the dangerous threat it poses, our country cannot be the victims [sic] of horrendous attacks by people that believe only in Jihad.'"

The other paragraph smartly summarized the reaction Trump has provoked:

"Leaders from across the political spectrum have condemned this policy, saying that banning members of an entire religion from entering the country goes against everything we believe in as Americans. And it will make our country less safe by alienating the allies we need to fight ISIS."

Respondents were asked again whether they favored or opposed "the proposal to temporarily ban all Muslims who are citizens of foreign countries from entering the United States." As Heilemann indicated, the numbers barely budged. But in case of either paragraph, the invitation is to double down on one's initial response to a less provocative question. No one likes to be called out the chump, even online.

This leads to a third caveat: the difficulty of controlling for the issue being tested. In the case of the first paragraph, were respondents truly still thinking about the "proposal," or were they reacting to "dangerous threat," "horrendous attacks" and "Jihad"? In the second, could the key phrase be "Leaders from across the political spectrum"--the embodiment of the Washington elite whose rejection is an important element of Trump's appeal, but one completely separate from his nativism?

This is what makes poll design so difficult: you only get one unpolluted response, particularly on a given subject. All answers that follow are prejudiced by what went before. It would be interesting to see a poll in which the rhetorically charged questions were sequenced first. Without one, we cannot be certain just what biases have been served, avoided, or even measured.

I have my own biases. As someone who has criticized Trump since before his inclusion in the first Republican debate, I crave analysis that keeps hope alive for the John Kasichs, Chris Christies and Marco Rubios of the world. My greatest concern is that this particular poll, carelessly interpreted, has led headline-writers the world over to overstate both the depth of commitment to Trump's policy among those who regularly vote Republican, and their embrace of his most caustic rhetoric.

This is not to say that Purple Strategies has crafted a bad poll. In fact, they've provided evidence that the feedback loop that has driven Trump's entire campaign has been internalized by many of his supporters: provocation - challenge - reinvest in the provocation. It is intuitive, the fight-or-flight instinct as flash-poll. But commentators should recognize that if reported uncritically, polls like this do not just measure the appeal of Trump's vicious rhetoric: they can help accelerate its acceptance.

My own bias against that is yuuuge.