How Reliable Are Polls About Religion? Princeton Professor Tries To Find Out

(RNS) The uses — and abuses — of public opinion polling made headlines this week after Fox News claimed it relied on surveys to select 10 of 17 Republican presidential candidates for its prime-time nationally televised debate Thursday (August 6).

Pundits howled and poll jargon flooded the media.

But political polling isn’t the only arena where scholars and the public have become  “fed up, frustrated and angry” at surveys that one leading sociologist claimed are of “dubious value or validity.”

Faith in religion surveys can be shaken too, said Robert Wuthnow of Princeton University.

He charges that the image of U.S. religion created by pollsters is too often inaccurate, shallow and misleading. A steady parade of surveys on faith and values misses the depth and nuance of American religious life while making puffed-up claims for credibility even as the rate of response falls to record low levels.

These poll findings are often misused by the media and misunderstood by the public, Wuthnow writes in his new book, “Inventing American Religion: Polls, Surveys, and the Tenuous Quest for a Nation’s Faith.”

Major pollsters and top experts on religion surveys agree — to a degree — as they study advance copies of the book (in stores Sept. 1). The first chapter appears in the conservative religion journal First Things.

Religion pollsters bow to Wuthnow’s standing in their field and they share some of his concerns. But, they added, their surveys are transparently executed and methodologically sound, and they offer revealing glimpses on how Americans see themselves.

”Polls and surveys are blurry portraits of reality but without them we would be nearly blind,” said Mark Gray, senior researcher for the Center for Applied Research in the Apostolate, which focuses on Catholic data.

Gray said the polling community is already giving serious attention to concerns Wuthnow raises. But to trash polling as we now know it “would set us back to the early 20th century when we just had anecdotes about some people’s religious behavior,” Gray said.

Alan Cooperman, director of religion research for the Pew Research Center, said pollsters “are not responsible for everything the public does — or should — know. And we don’t pretend we are the only or best way.”

But the effort is worthwhile, he said: When ancient rabbis examined a passage in the Book of Deuteronomy about the first census, Cooperman said, they concluded, “We count things that we care about.”

Much of the concern shared by Wuthnow and pollsters is about response rates: The percentage of people called on who agree to participate in a poll has slid to 10 percent or less. This raises questions about whether the participants represent a valid cross section of public opinion.

Wuthnow’s book cites as an example a major survey on Jews in America conducted by Pew Research in 2013. Pew went to great cost and four months of effort to elevate the response rate from 9 percent to 16 percent. Wuthnow scoffs: Was that “truly the best a commercial polling agency could do?”

His answer to the problem of pumping up the numbers? Pollsters should save their money.

“Fewer and better surveys is a reasonable way to go,” he said.

What are the chances of that?

“Pretty low!” said Wuthnow, laughing. “Maybe they can’t get much better and we may be at a point where not much more can be done except for them to be as transparent as possible in their press releases about the rate of response and the margins of error.”

 (Margin of error — the minimum and maximum percentage points that sampling error could introduce in poll findings — was a leading reason pundits objected to Fox News’ candidate debate selection process. Given the margin of error, typically plus or minus 3 percentage points in national polls, Ohio Gov. John Kasich, who was invited to the prime-time debate, was statistically tied with Texas Gov. Rick Perry, who has been relegated to the second-tier candidates’ dinner-and-drive-time debate.)

Cooperman nonetheless bit back at Wuthnow’s criticisms of religion survey response rates.

Response rate is not a reliable indicator of survey quality, based on three major studies Pew has conducted, Cooperman said.

The studies looked at whether the respondents in low-response surveys were different or similar to participants in government surveys with very high response rates and with the General Social Survey, conducted biennially by the National Opinion Research Center.

They turned out to be “very similar,” said Cooperman.

David Kinnaman, president of the Barna Group, shares Wuthnow’s concern that the public is skeptical of poll results when anyone with a Twitter log-on or a Facebook feed can “poll” his or her followers and tap into “everyone who already agrees with them.”  

The Barna Group is unique among mainstream pollsters for using a complex set of theological measures to categorize respondents rather than letting people loosely label themselves as “evangelical” or “Christian” or “born-again.”

Wuthnow said the Barna Group’s attention to nuance is rare. He worries about quantifying faith.

“It gives an aura of precision to say that Pew and Gallup show 90 percent of Americans believe in God, but the numbers say nothing about what people believe about God or how they experience God in their lives.”

Wuthnow would like to see pollsters turn to more open-ended questions and focus groups where people can push back against the pollster’s language and define their own terms for their religious life. Indeed, he thinks that exposure to decades of pollsters’ questions and media-touted findings have shaped the vocabulary of how people respond.

His prime example: Asking people if they are evangelical or born-again as a single category of identification. This has trained the public to think, incorrectly, that these very different theological terms are one and the same, said Wuthnow.

That makes the answers actually meaningless in religious terms even if they are useful politically as a pointer to “who is more likely to vote Republican,” he said. 

 Robert Jones, CEO of the Public Religion Research Institute, is not so fast to claim that much power for polling.

“I would like to think that polling has that much cultural power but there are all kinds of forces at play behind the ways people respond. More diverse lifestyles, diverse families, diverse cultures are at play than when the white nuclear family was the cultural framework,” said Jones.

It’s an open question how much all the introspection among pollsters and academics will lead to changes that satisfy Wuthnow. Ultimately, in his view, religion isn’t encapsulated in sound bites.

“It’s all about stories and narratives and the imagination,” he said, paraphrasing the late Catholic priest and sociologist Andrew Greeley. “We don’t get at these with surveys.”

So, Wuthnow offered advice to consumers of religion data: “If you read somewhere that Harry Potter is more popular than Jesus, be amused, but don’t take it seriously.”

Also On HuffPost: