PPP Polls Shows Why Issue Polling Is So Unreliable

One of the favorite shticks of Democrat pollster Public Policy Polling (PPP) is to ask questions designed to make Republican voters look bad. This kind of “troll polling” flatters all the usual sorts of people who love to laugh at what yokels the GOP’s supporters are, and as yet no Republican-leaning pollster has gotten into the regular business of giving Democrats a taste of the same medicine. If the last few years have taught us anything, it’s not to trust individual polls that can’t be checked against a polling average, but by definition these are all one-off polls. But there’s a deeper issue here that the latest PPP trolling question illustrates: that average Americans are far too trusting of pollsters, and the ability of pollsters to exploit that trust shows why polling on individual issues is untrustworthy.

Here’s the latest poll question that has PPP’s followers floating on a cloud of smug this morning:

30% of Republican primary voters nationally say they support bombing Agrabah. Agrabah is the country from Aladdin. #NotTheOnion

— PublicPolicyPolling (@ppppolls) December 18, 2015

Over six thousand Retweets at this writing! A flavor of what PPP is trying to accomplish comes from the following Tweets:

30% of Republicans want to bomb a country from a Disney movie and it’s PPP’s fault https://t.co/CdYZar3jAs

— PublicPolicyPolling (@ppppolls) December 18, 2015

Agrabah is trending. Thank you @ppppolls pic.twitter.com/w56oqwb36E

— Mai El-Sadany (@maitelsadany) December 18, 2015

If you look at the overall breakdown on PPP’s poll question and how Democratic voters answered the same question, Republicans don’t actually come out looking quite as bad as the Tweet headline suggests:

We asked the Agrabah question to Dem primary voters too. They oppose bombing ‘it’ 36/19, while GOP supports bombing ‘it’ 30/13

— PublicPolicyPolling (@ppppolls) December 18, 2015

If you’re keeping score at home, that means 55% of Democrat voters were willing to express an opinion on bombing a fictional country, compared to 43% of Republicans. PPP is dining out on the 30-19 edge in Republican voters who said “yes,” but if you take this poll seriously, the 57-45 edge in Republican voters who were unwilling to answer a question with an egregious falsehood about world events embedded in its premise seems to cut in the opposite direction than what PPP is trying to accomplish here.

But really, you should not take these troll polls seriously, and in fact they should teach educated poll consumers to be skeptical about all issue polling. Why did 55% of Democrats and 43% of Republicans answer a question about bombing a fictional country? Partly, one assumes, because it didn’t occur to them that the pollster would take advantage of them by asking a question that assumed facts that do not exist. Partly because people in general do not like to admit there are things they do not know. Partly because people do assume there are all sorts of little countries out there they have never heard of, a fair number of which (e.g., former Soviet republics, parts of the old Yugoslavia, breakaway African states) didn’t exist 20 or 30 years ago, and that some of these are unstable places that may house the occasional wretched hive of scum and villainy. And partly because answers to issue polling questions tend to vary a lot by what the pollster says before asking them:

In this particular case, the Agrabah question was Question 38 on a 41-question poll. If you’ve ever taken a poll, 38 is a lot of questions – you’re getting impatient, the kids may be yelling for you, but if you are a good-natured sort and want your voice heard and you’ve already invested several minutes of your time, you’re determined to stick it all the way to the end by this point. (We don’t know how many people hung up before they got to the end). Let’s look at the issue questions leading up to this:

Question 29 Do you support or oppose requiring a criminal background check of every person who wants to buy a firearm? (6% were not sure)

Q30 Would you support or oppose a bill barring people on the terrorist watch list from purchasing a firearm? (7% were not sure)

[Q31 was a mimumum wage question; only 2% were not sure]

Q32 Would you support or oppose banning Muslims from entering the United States? (21% were not sure)

Q33 Do you believe that thousands of Arabs in New Jersey cheered when the World Trade Center collapsed on 9/11 or not? (29% were not sure)

Q34 Would you support or oppose shutting down mosques in the United States? (26% were not sure)

Q35 Would you support or oppose creating a national database of Muslims in the United States? (17% were not sure)

Q36 Do you think the religion of Islam should be legal or illegal in the United States? (21% were not sure)

Q37 Looking back, do you support or oppose the policy of Japanese Internment during World War II? (23% were not sure)

Q38 Would you support or oppose bombing Agrabah? (57% were not sure)

You’ll notice first that the number of people who didn’t feel qualified to answer questions about gun ownership and the minimum wage was pretty low, but rose as they moved into the next set of questions and then abruptly more than doubled when they got to bombing the land of genies and magic carpets. You’ll also notice how the poll let into this with a series of questions all tied around terrorism and Islam, so respondents were primed to expect that a question about bombing some obscure Arab-sounding place was related in good faith to the questions that came before it – that bombing Agrabah was a thing that our leaders were seriously discussing.

You’ll also notice if you dig into the crosstabs that the most liberal (31%) and youngest (46% under age 45) and male (51%) poll respondents were the least willing to refuse to answer (all this is among Republicans; PPP hasn’t yet released the poll of Democrats). Interestingly, aside from George Pataki (who polled at 0% so his support can’t have been more than 1 or 2 people), the two candidates whose supporters were least likely to answer the question were Ben Carson and Mike Huckabee – 76% of Carson’s supporters declined to answer it.

But the crosstabs are beside the main point, which is that people are willing to tell pollsters all sorts of things about which they do not actually have anything like a fixed opinion, let alone an informed one. I very much doubt if PPP polled a single person who went into that call with an opinion about bombing Agrabah, and I doubt very many of them continued to have an opinion the next day. A machine asked them to press a button, so they just took their best guess.

And if you read every issue poll from today forward with that in mind, you will realize how much of the issue polling that gets published is no more useful or predictive than knowing people’s opinions about bombing a place that exists in a Disney cartoon. PPP may have been looking to discredit Republican voters, but it really did more to reveal the problem with the trustworthiness of its own industry.

The post PPP Polls Shows Why Issue Polling Is So Unreliable appeared first on RedState.