Beware: Survey Questions About Fictional Issues Still Get Answers

WASHINGTON -- As they do on many obscure policy issues, Americans polarize sharply along partisan lines when they learn that President Barack Obama supports a repeal of the 1975 Public Affairs Act. Republicans are over 10 times more likely to disagree with the president on this issue than are Democrats.

There's one striking problem here: The 1975 Public Affairs Act does not exist.

The Huffington Post and its polling partner YouGov recently repeated a classic survey experiment that highlights both the tendency of Americans to divide along partisan lines on almost any issue and the dangers of over-interpreting the results of political poll questions.

Political scientist George Bishop and his University of Cincinnati colleagues were the first to ask about the fictional "Public Affairs Act of 1975." In a series of surveys that polled national samples about similarly fictitious or otherwise unknown legislation, they found 20 to 40 percent of Americans were willing to offer opinions on laws they have never heard of.

Their conclusion: Some survey respondents feel social pressure to answer the question asked, no matter how confusing or unfamiliar, so they may form new opinions on the spot, drawing cues from the words or concepts conveyed by the question.

The initial survey experiments were conducted by telephone or in person. While respondents could have volunteered that they "don't know," interviewers did not offer that option explicitly. Suspecting that some individuals were unwilling to admit ignorance, Bishop and his colleagues tested an alternative version that offered a third option: "haven't thought much about this issue." Giving respondents an out reduced the number of those expressing an opinion on the fictitious Public Affairs Act on one experiment from 24 to 4 percent.

The new HuffPost/YouGov survey's approach and the results obtained fall somewhere in between. YouGov's surveys are conducted online without an interviewer and typically offer "unsure" as an explicit response option on most questions.

In answer to the initial question, 15 percent offered an opinion -- 9 percent agreed that the fictitious law should be repealed and 6 percent disagreed -- while 85 percent chose "unsure."

That question, which began by informing respondents that "some people say the 1975 Public Affairs Act should be repealed," produced just a hint of partisanship. Democrats were only slightly less likely to agree than independents or Republicans that the law should be repealed, although given the sample size of 334 respondents, the differences between the subgroups were far too small to be considered statistically significant.

But the two alternative questions, each asked of separate samples of more than 600 respondents, made the partisan connection more explicit. When informed that "President Obama says" the 1975 Public Affairs Act should be repealed, 39 percent of Republicans expressed disagreement, compared to just 3 percent of Democrats. When informed that "the Republicans in Congress" want to repeal the act, 28 percent of Democrats disagreed compared to just 4 percent of Republicans.

Finally, we asked all of our respondents what they thought the Public Affairs Act was about. Far more thought it was "an attempt to reduce lavish fundraisers by elected officials" (11 percent) than believed it to be "a requirement that all members of Congress hire public affairs firms" (3 percent) or "a ban on extramarital affairs by members of Congress" (3 percent). The rest either opted for "none of the above" (20 percent) or were uncertain (64 percent).

The lesson here is straightforward: Many poll respondents will offer opinions on issues they know little or nothing about, making it difficult to distinguish pre-existing opinions from reactions formed on the basis of the words of the question. Poll respondents will find it even easier to offer an opinion when informed where well-known political leaders stand on the issue.

It is always best when interpreting survey results to consider how familiar Americans are with the issue, how many are reluctant to offer an opinion and how those who are closely following an issue differ from those who are not.

The poll was conducted March 29-30 and April 5-6 among a total of 1,664 adults using a sample selected from YouGov's opt-in online panel to match the demographics and other characteristics of the adult U.S. population. Factors considered include age, race, gender, education, employment, income, marital status, number of children, voter registration, time and location of Internet access, interest in politics, religion and church attendance.

The Huffington Post has teamed up with YouGov to conduct daily opinion polls. You can learn more about this project and take part in YouGov's nationally representative opinion polling.