Election Polling Mea Culpas

[Kyle]

So I've been reading the mea culpas that pollsters have been putting out about how the election polls were so off. Some are good (the Washington Post guy, on a podcast, was particularly good) and a lot are poor ("Sure we got a lot wrong, but we got a lot right!").

Personally, it makes me angry. I bought into the polling. I checked in with 538 every day. I actually believed that there was no way that many polls could be off. I actually believed that they had taken steps to adjust for the Trump factor that screwed up the 2016 polls. So a lot of my anger is that I feel stupid for falling for this. Again. But a lot of my anger is that I defended the polling and now, because they screwed up again, these polls are feeding into the bullshit conspiracy/fake news/stolen election horseshit. It's all infuriating.

A special "Burn in Hell" to 538. They didn't apologize at all (unless I missed it). They just said, "Hey, we're not pollsters. We just report the polls." (Mike pointed me to this the first time I complained about this just after the election. This isn't a gripe about Mike.) But that's so much bullshit. When you report on all the polls and then interpret those polls... and then put odds on how an election turns out based on those interpretations... and run 50,000 simulations... and then give your odds and predictions based on those simulations... you don't get to say, "Not my fault!" Just like the pollsters don't get to say, "We just report what people tell us! Not our fault!" 538 and pollsters interpret and distil the raw data for us to understand. They interpret it. They tell us which are more reliable than others. To come out and go, "Oh that sucks. You pollsters need to get your shit together," is just so much garbage. Fuck you, 538.

[poorpete]

To do a tiny bit of defending, their rating of likelyhood included a "what if there's a large polling error" -- though I think that included a large polling error that'd benefit Biden - I still think polling has a lot of explaining, especially in relation to the Margin of Error. It's pretty clear that polling is shit now for anything relatively close, that due to the shedding of land-lines, it's become increasingly hard to get a gauge on the electorate. They weren't wrong that their fixes for 2018 seemed to do the trick, but it was a mirage, there seems to be a big issue when trying to figure out who is a likely voter, and to get those voters to participate in public polling.

Here's another defending of the numbers guys: apart from the polls, they were EXTREMELY accurate at analyzing the real number, you know the votes. While betting places flooded with Trump pulling another upset in the late evening of election day, the numbers nerds were more bullish on Biden than nearly anyone. And then after a day or so, they were a lifeline whenever my fears got the best of me, there there, Pennsylvania's moving towards Biden at a great pace, and Arizona is moving to Trump not enough to make a difference.

Proof maybe that these are very smart folks with numbers, but very dumb when the numbers are bad.

[Kyle]

To do a tiny bit of defending, their rating of likelyhood included a "what if there's a large polling error" -- though I think that included a large polling error that'd benefit Biden - I still think polling has a lot of explaining, especially in relation to the Margin of Error. It's pretty clear that polling is shit now for anything relatively close, that due to the shedding of land-lines, it's become increasingly hard to get a gauge on the electorate. They weren't wrong that their fixes for 2018 seemed to do the trick, but it was a mirage, there seems to be a big issue when trying to figure out who is a likely voter, and to get those voters to participate in public polling.

This is actually the source of a lot of my anger about this. I wish I had a link to the podcast the Washington Post guy was on, because he really laid it bare. He basically talked about the land lines and how they were having to contact a hundred fold more people just to get enough respondents compared to 15 years ago. And he also admitted (and apologized) that they knew these methods made the data inherently unreliable, but that pollsters think they can make the right adjustments and interpretations to the data. But his point is that it's a poisoned well when the data is that bad.

Here's another defending of the numbers guys: apart from the polls, they were EXTREMELY accurate at analyzing the real number, you know the votes. While betting places flooded with Trump pulling another upset in the late evening of election day, the numbers nerds were more bullish on Biden than nearly anyone. And then after a day or so, they were a lifeline whenever my fears got the best of me, there there, Pennsylvania's moving towards Biden at a great pace, and Arizona is moving to Trump not enough to make a difference.

Agreed 100%. And while some of that was based on exit polling, most of it was based on experience.

[Stan]

One thing pollsters rarely bother to explain is that the margin of error still assumes perfect sampling, or sampling that is perfectly representative of the population. The better polls take into account what they can, like over representing results from ages and genders that got fewer responses. But they can't take into account that those who respond to polls might be inherently different than those who do, because they have no data from people who don't respond. It's not just lack of landlines, phone spam makes a large number of people (including me) not answer unknown numbers.

[DMDarcs]

Totally agree with Stan. In the weeks moving up to the election, I was contacted by phone for polling surveys eight different times. Each time was with different polling groups, and each time had different biases. (Example: one poll asked me if I was more likely to vote for one candidate based on their track record with stimulating the economy based on "job creation" and if I was less likely to vote for one candidate based on their push to give ex-felons the "right to vote".) For the first several, I answered very honestly, and didn't pull punches. But on the sixth one I've gotten in a week - it started to get annoying. When I told the pollster I didn't really feel like doing this, they assured me it would only take me a few minutes, and I told them that I wasn't going to necessarily be accurate with the questions they were asking. You catch someone at the wrong time, they're not going to give you an accurate answer. And the questions that they ask don't allow for nuance in the answers. Quite often, I would give an answer, and it wouldn't fit in one of their boxes. I told them to just figure out whatever box made the most sense for them based on my answer.

[Phoebe]

After the gruesome surprise of 2016, I wasn't willing even to read any of that information or put any stock in it, so while I was happy to see Biden with a huge lead, it did not come as a surprise that it wasn't a landslide or the blue wave or whatever. It puzzles me that this industry, driven as it is by a desire to provide accurate insights and predictions, is having such a hard time making the necessary corrections. Do we lack basic data across the board that is necessary to understanding today's voters? What impact is the slow-motion disaster of the Census going to have on this? Who still has landlines and who does not?

[Mike]

Polls are broken and have been for a while. Trump makes them extra broken.

Quick defense of polling: Both 2016 and 2020, the polls were off by absolutely normal margins compared other presidential election years. The errors were not unusual. The problem is that both times, they underestimated Trump, and when the races have all become hyper-partisan and extremely close, those normal 3-5% errors make huge differences. Note that in 2018, the polls and 538's estimates were pretty close to the final tallies. But that's not a presidential race with a fascist on the ballot.

The problem I have is that with our current technology and analytical tools, our estimates, even with flawed polling, should be getting better and they are not. And I don't see anyone with a plausible solution that will make it better.

The problem I have with 538 is that Nate Silver is an asshole. He has an awesome staff of insightful people. I enjoy 538 for their political analysis outside of sheer number-crunching forecasting, but now that ABC News has fired Clare Malone from the 538 team, I don't know that I'll be going back for that even. I'm disgusted with Nate Silver for a lot of the reasons listed above. True, he doesn't conduct the polls, he merely compiles them. But there are A LOT of judgment calls he puts into his forecast model, so his analysis MEANS SOMETHING. Him claiming, "Hey, I just crunch the numbers," is just as disingenuous as all the times John Stewart said, "Hey, I'm just a comedian." Nate's been dropping a lot of F-bombs on his podcast lately when responding to criticism, and it's pretty gross.

[Phoebe]

The point about 2018 is good. If it was all messed up, why do they get some things much closer to right? I also wonder how much of this (not 538 specifically, but the overall narrative that develops) happens because people are writing speculative stories about Democrats retaking the Senate, which always seemed like a possibility but not the likely outcome, and then the reality of deflated-sails seems much more deflated than it should? Anyway, it would be interesting to hear more about the measures they are taking to improve accuracy as people and their habits change over time.

[FlameBlade]

This is like weather forecasting. You set expectations based on available information. When weather event, weather forecasters should explain where things are going off the expectations so people can adjust accordingly. Same with pollsters. If something is wrong with polls, then pollsters should be able to describe what is off about their models. The big mistake pollsters make is arrogance, and using it to beat others down their throats. Polls are just models, and pollsters need to be honest about what it really means.

Nate Silver is insufferable asshole. Clare Malone is awesome.

[Mike]

But now the polling for the Georgia run-offs was pretty much dead on. This means that pollsters all have an out by saying that it's just Trump being on the ballot that throws things so out of whack... back to business as usual. And there is maybe some validity there, but it is far from all of the story.

[Kyle]

But now the polling for the Georgia run-offs was pretty much dead on. This means that pollsters all have an out by saying that it's just Trump being on the ballot that throws things so out of whack... back to business as usual. And there is maybe some validity there, but it is far from all of the story.

Yeah- but this may actually be true. I'm also not convinced that the next "Trump" will have the same sway as Trump did.