Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Arsenic Lupin
Apr 12, 2012

This particularly rapid💨 unintelligible 😖patter💁 isn't generally heard🧏‍♂️, and if it is🤔, it doesn't matter💁.


Two American elections running, pollster predictions have been substantially off, and not just at the Presidential level.

I have a question. When phone polls began, you only got phone calls you were somehow connected to: family, friends, a business or two. A call from somebody else was novel and might be interesting. There was no reason to (or excuse for!) ignoring a phone call.

It's 2020. Spam is inescapable, and a lot of spammers claim to be running a poll as a way to hook the fish. A couple of generations don't bother with land lines. Many (most?) people under sixty screen calls, if they even accept calls without a text message first. And there's no reliable caller ID.

1. Is it possible to tune phone polls so that they still reach a representative population?
2. Do people still tell the truth to pollsters, in whatever method they're reached?

In short, are phone polls doomed? If not, why not, and if so, what comes next?

Adbot
ADBOT LOVES YOU

Zogo
Jul 29, 2003

Arsenic Lupin posted:

1. Is it possible to tune phone polls so that they still reach a representative population?
2. Do people still tell the truth to pollsters, in whatever method they're reached?

In short, are phone polls doomed?

Yes, but you can't force someone to answer the call.

No, because people are paranoid for a variety of reasons. Scammers are pervasive. And the avoidance of ostracization.

It seems that way.

Arsenic Lupin posted:

...if so, what comes next?

https://www.youtube.com/watch?v=eCsT_ocYDZ0

Jayne Doe
Jan 16, 2010
I imagine there might be a shift to polling via text message instead. I certainly got a lot of "poll: who will you vote for in the 2020 presidential election?" texts this election cycle. (from what appeared to be a legitimate organization, when I looked into it)

Arsenic Lupin
Apr 12, 2012

This particularly rapid💨 unintelligible 😖patter💁 isn't generally heard🧏‍♂️, and if it is🤔, it doesn't matter💁.


Jayne Doe posted:

I imagine there might be a shift to polling via text message instead. I certainly got a lot of "poll: who will you vote for in the 2020 presidential election?" texts this election cycle. (from what appeared to be a legitimate organization, when I looked into it)

But that still has problems 1 and 2: who replies, and do they tell the truth?

PT6A
Jan 5, 2006

Public school teachers are callous dictators who won't lift a finger to stop children from peeing in my plane
Yeah, my parents always answer phone polls because they're old and bored, but I don't think they're representative of anything in general. I answer random numbers but as soon as I hear a computerized voice I hang up so who knows if they've attempted to poll me or not? With a live operator actually on the line, I do answer some, but I've never got a political poll that way.

Blue Footed Booby
Oct 4, 2006

got those happy feet

Back when I had a landline I got polls all the time. Once I got rid of it I never got a poll again.

Unless the government figures out and implements a way to completely end spam calls or someone comes up with a way to poll that's only accessible to legitimate organizations, polls are going to keep getting worse.

KOTEX GOD OF BLOOD
Jul 7, 2012

I highly recommend this article from the American Association for Public Opinion Research on the 2016 polling error to gain more insight into this issue.

Arsenic Lupin
Apr 12, 2012

This particularly rapid💨 unintelligible 😖patter💁 isn't generally heard🧏‍♂️, and if it is🤔, it doesn't matter💁.



This bit is really interesting:

quote:

Adjusting for over-representation of college graduates was critical, but many polls did not do it. ... Furthermore, recent studies are clear that people with more formal education are significantly more likely to participate in surveys than those with less education.

KOTEX GOD OF BLOOD
Jul 7, 2012

A lot of the issues with phone polling are just issues with polling. People are idiots. They give responses to questions they know nothing about. They give responses that aren't what they actually believe, but because they think they are the "right" answer, or the answer that they think the survey administrator somehow wants them to give. They change their responses based on small changes in question wording. They are easy to sway with big changes in question wording. I have watched consultants deliberately design polls and surveys poorly so as to produce a result that is preferable for their client, and this is easy to do because people act like idiots.

On a relative basis, presidential polling at the national level is fairly reliable, even with the vagaries of phone polling. Aggregate national polling in both 2020 and 2016 estimated the popular vote with a fair degree of accuracy. You're dealing with a huge sample size and asking them a simple question about two people who show up on TV all the time, one versus the other. State-level polling is a lot more difficult, and unfortunately for the presidential election, that's all that matters due to the electoral college, and in the swing states you are often dealing with the type of people who don't pick up the phone, and in both 2016 and 2020 this helped to skew state-level polls. It is hard, but not impossible, to overcome this effect through weighting. That being said, there is a general consensus among political scientists that there is no "shy Trump voter" effect.

Attitudinal polling (polling about issues) is awful, and to a certain extent you can just treat it as bullshit for the reasons stated above. Huge long-term academic studies like the ANES and CCES are going to be more reliable, but still. The absolute worst is the "presidential approval" question. Gallup's, for instance, is "Do you approve or disapprove of the way ... is handling his job as president?" This sounds straightforward but is actually super vague! People interpret it in all kinds of ways.

I don't think the practice of polling is itself doomed, but as I have warned in D&D again and again since 2016, estimating the Electoral College outcome based on state-level polling is as of now a fool's errand.

Arsenic Lupin
Apr 12, 2012

This particularly rapid💨 unintelligible 😖patter💁 isn't generally heard🧏‍♂️, and if it is🤔, it doesn't matter💁.


As it happens, there's an article by pollster David Hill in today's Washington Post!

quote:

When I first undertook telephone polling in the early 1980s, I could start with a cluster of five demographically similar voters — say, Republican moms in their 40s in a Midwestern suburb — and expect to complete at least one interview from that group of five. I’d build a sample of 500 different clusters of five voters per cluster, or 2,500 voters total. From that number, I could be reasonably assured that 500 people would talk to us. The 500 clusters were designed to represent a diverse cross-section of the electorate.

As the years drifted by, it took more and more voters per cluster for us to get a single voter to agree to an interview. Between 1984 and 1989, when caller ID was rolled out, more voters began to ignore our calls. The advent of answering machines and then voicemail further reduced responses. Voters screen their calls more aggressively, so cooperation with pollsters has steadily declined year-by-year. Whereas once I could extract one complete interview from five voters, it can now take calls to as many as 100 voters to complete a single interview, even more in some segments of the electorate.

And here’s the killer detail: That single cooperative soul who speaks with an interviewer cannot possibly hold the same opinions as the 99 other voters who refused.
...
I offer my own experience from Florida in the 2020 election to illustrate the problem. I conducted tracking polls in the weeks leading up to the presidential election. To complete 1,510 interviews over several weeks, we had to call 136,688 voters. In hard-to-interview Florida, only 1 in 90-odd voters would speak with our interviewers. Most calls to voters went unanswered or rolled over to answering machines or voicemail, never to be interviewed despite multiple attempts.

The final wave of polling, conducted Oct. 25-27 to complete 500 interviews, was the worst for cooperation. We could finish interviews with only four-tenths of one percent from our pool of potential respondents. As a result, this supposed “random sample survey” seemingly yielded, as did most all Florida polls, lower support for President Trump than he earned on Election Day.

Dante
Feb 8, 2003

the short version of this is that it's getting increasingly difficult to get a representative survey of the population to the point where many of the non-public polls aren't done using phone polling at all. This isn't because general responsiveness has changed (at that point it's merely a cost increase in the numbers dialed), but because while the percentage of people who answer polls has declined it has declined unevenly (non-response bias). It appears that rates about those that score low on social trust in the GSS have plummeted much more than others (appears to be a more correlated metric than college_ed), while also some parts of the electorate has become more engaged and started answering polls at an increased rate. Another way of saying it is that not responding to phone surveys is now correlated to partisanship while it wasn't earlier. Basically the old way of doing phone polls are dead because response rates have gotten so uneven for subgroups you can no longer reliably statistically control for it and from now on you'll probably have to do more expensive panel type surveys. Pew research as one of the major ones for example have almost dropped phone polling entirely. This might seem like bad news for polling in general, but phone polling isn't really that old - being mostly a more cost-effective way of polling that started getting heavily used in the early 1980s. The super-reliable high rate of response surveys like GSS still hire lots of volunteers and do it the expensive old-fashioned way of live interviews! (note: not entirely free of mode effects due to social desirability bias etc).

Dante fucked around with this message at 07:06 on Dec 1, 2020

Adbot
ADBOT LOVES YOU

Devonaut
Jul 10, 2001

Devoted Astronaut

fivethirtyeight put out a podcast earlier this week with lots of discussion about phone polling among other poll topics

https://fivethirtyeight.com/features/politics-podcast-how-to-make-polls-better/

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply