Though the experiment dealt with an issue that , I'm writing about it now because it demonstrates how questions about specific policy plans can produce misleading results about the public's views of the broader issue - a classic case of not seeing the forest for the trees.
SurveyUSA Experiment
of phrasing the bailout question, and each one found mixed to slightly negative results. But then two follow-up questions starkly contradicted these results to suggest a clear majority of the public was supportive of the bailout efforts.
While the SurveyUSA experiment tested four different ways of wording the bailout issue, three are rigorously comparable, and so I mention them first. I'll come back to the implications of the fourth version later in this post.
Each of the following questions was asked of a split sample of just over 500 respondents:
- The government may invest billions to try and keep financial institutions and markets secure. Do you think this is the right thing for the government to do? The wrong thing for the government to do? Or, do you not know enough to say?
- The government may spend billions to bail-out Wall Street. Do you think this is the right thing for the government to do? The wrong thing for the government to do? Or, do you not know enough to say?
- The government may spend billions to rescue Wall Street. Do you think this is the right thing for the government to do? The wrong thing for the government to do? Or, do you not know enough to say?
Results of Three Versions of Bailout Question
"Right or Wrong Thing to Do?"
Was the Public Ambivalent?
With these results, one would have to conclude that the public was at best ambivalent toward a bailout or rescue of Wall Street. Given the sample sizes, Form A results are significantly different from those in Form C, suggesting "rescue" is the most negative way to phrase the issue and "invest" is the most positive way.
But then come two additional questions that throw a completely different light on the issue. The question below taps into a general feeling about the issue, and finds that people seem to want Congress to do something - and that they're more afraid Congress will do too little than too much.
More Afraid Congress Will Under-react Than Over-react
Little
The problem with comparing the above question to the other three is that this one doesn't explicitly allow for "no opinion," while the first three questions do.
Public Supports Bailout
But the next question is comparable in offering an explicit "don't know" response, and it also suggests that a clear majority of the public wants Congress to do something.
Should Congress Support SOME Economic Rescue Effort?
Note that despite offering an explicit "or don't you know enough to say" option, this question shows a clear majority in favor, with only 30 percent opposed.
Had we depended on only the first three questions, which were asked of separate split samples, for our understanding of the public, we might well have concluded that whether it was "bailout" or "rescue" or "invest," the public was either evenly divided about a bailout or leaning against such an effort. But these last two questions suggest that less than a third of the public was opposed to an economic rescue/bailout plan in principle, while a clear, though small, majority (54 percent) of people were in favor.
New Complications
When we look at the results of the fourth version of the split sample experiment, we find even more support for some type of rescue plan. This version included three substantive options (compared with just two for the other three versions - which is why I'm treating this question differently from the first three versions), followed by the explicit option of not expressing an opinion.
Fourth Version of Bailout Question
Plan
This version shows the least support for the current plan, but its second option - to pass a "different" plan - suggests that more than seven in 10 respondents are in favor of some rescue plan (31 percent the current plan, plus 40 percent for a different plan). These results also suggest that only 13 percent (rather than 30 percent as shown in Table 3) are opposed to some kind of effort.
Important Lessons
These results, based on a telephone survey in the San Francisco Bay Area, demonstrate how variable the survey results can be - even when the survey is about an issue that is the subject of much media attention. The first three versions of the bailout question could reasonably be interpreted to suggest that the Bay Area public was either ambivalent or negative about any bailout plan, while the question reported in Table 3 gives the opposite impression - that the same public was in fact looking for some action by the federal government.
The important lesson: Sometimes, asking about specific plans can blind us to the larger issue of whether some type of action is still desired.
The last question reaffirms that when questions have two options in favor of a policy (pass the current plan, or pass a different one) and one against (take no action), the "opinion" that is measured can be quite different from when a question has just one option in favor of a policy and one opposed.
The last question can also be interpreted to have two options against the "current" plan - only one option says pass the current plan, while two options are against it (pass a different plan, and take no action).
This doesn't mean that three options should never be offered. The example does reaffirm, however, that the more options that are offered in general, the less similar the results will be to questions that offer only two options.
Note: My thanks to SurveyUSA's Jay Leve for pointing me to his very insightful experiment.