How did Trump win 2016? It appeared to turn the world upside down, seemingly defying polls and reason. What happened? Was it really surprising or were we misled?
The game plan will be first to examine a few popular myths, eliminate these explanations, then examine the results of 2016. By articulating why 2016 was surprising, we will find the factors responsible for the outcome.
Executive Summary: It's all Gary Johnson's fault.
Remark (On Obama-Trump Voters). One plausible explanation for Trump's victory is the shift of Obama voters of 2012 who came to vote for Trump in 2016, the so-called "Obama-Trump voter". This is worth mentioning, only in passing, because the media has an irrational fascination with such voters.
Myth #1: Shy Trump Supporter
A popular conjecture is that a subpopulation of Trump supporters believed supporting Trump was "socially undesirable", hence would not publicly acknowledge backing Trump. Alexander Coppock conclusively tested this theory, and found no evidence of a shy Trump supporter existing. The predictive models with shy Trump supporters make statistically indistinguishable predictions from those without shy Trump supporters.
Andrew Gelman independently reckoned the same conclusion along different lines. Also,
Gelman reasons, Republican candidates outperformed expectations in the Senate races, which casts doubt on the model in which respondents would not admit they supported Trump; rather, the Senate results are consistent with differential nonresponse or unexpected turnout or opposition to Hillary Clinton. It is possible that the anti-media, anti-elite, and even anti-pollster sentiment stoked by the Trump campaign has been a part of the reason for the low response of Trump supporters in states with large rural populations.
Emphasis added.
It is worth remembering that Politico/Morning Consult released a poll, gathered online and via live phone calls, indicating despite different methodologies the different results show only a slight, not-statistically-significant difference in their effect on voters’ preferences for president.
In other words, it didn't matter if a respondent talked to a pollster on the phone (where shyness would prevent the respondent from announcing support for Trump) or if the respondent communicated online, the results were statistically indistinguishable.
Testing this hypothesis three different ways, and they all reach the same conclusion, seriously undermines the "null hypothesis" of the existence of "shy Trump supporters". We can discard this explanation as lacking empirical support.
Myth #2: Comey Ruined the Election
Hillary Clinton has personally blamed the election outcome on Comey's public announcement that he was re-opening the FBI investigation into Clinton's email servers the Friday prior to the vote. This explanation has become popular, presumably for the proximity of Comey's announcement to the perceived surprising loss. Neither evidence nor reason supports this misconception.
This claim is a contentious matter. But the Comey letter is the sort of mirage which our cognitive biases are susceptible to mistake for real. We must heed Thucydides's words (1.21), stressing the search for truth strains the patience of most people, who would rather believe the first things that come to hand.
Using rather rudimentary post-stratified modeling techniques, Chad P. Kiewiet de Jonge and Gary Langer have shown in fact Clinton began losing the projected electoral college the Tuesday prior to Comey's announcement. Why didn't anyone else have this insight? The other prognosticating psephologists used some combination of poll-weighting and a likely voter model, which would have missed it.1Nate Silver noted, regarding Comey's letter, As of Oct. 28, the polls-plus version of FiveThirtyEight’s forecast, which accounts for these factors, expected Clinton to lose a point or so off her lead before Election Day.
Silver's model did not detect the deviations which MRP modeling found.
The New York Times released a poll, released days prior to the letter, showing Trump ahead of Clinton by 4 points. Bloomberg/Selzer released a poll showing Trump ahead of Clinton by 2 points. Could we trust this poll? Well, FiveThiryEight has awarded Selzer & Co. an "A+" pollster grade (and similarly high marks for both Siena College and the New York Times).2As of July 3, 2019 and November 11, 2016 Selzer & Co. received an "A+". The New York Times, in collaboration with CBS, received the more modest "A-" grade, but Siena College won a solid "A". Although this is weak evidence supporting the claim Clinton began losing before Comey's announcement, the point we stress is this is a second approach to think about the matter.
Could it be that Comey's letter accelerated Clinton's decline? From post-stratified modeling based on polling released afterwards, there is insufficient evidence that Comey's letter impacted Clinton's standing. The MRP model using polling data with surveys held November 4–6 show Clinton recovering, albeit insufficient to win the election or recover significant ground. Such results challenge and undermine the claim Comey impacted Clinton at all.
The raw data, and refined statistics, both reach the same conclusion: Clinton began losing the election before Comey even spoke.
What Happened?
To better answer this question, we should first note what people expected. If, for each state, we take a rolling mean of the proportion of the vote for each party (bundling all "third parties" together into a single "third party"), then normalize the result per state (so we have proportions again), the result is precisely the proportion of the vote one might expect to have found on election day of 2016. The results may be described by the following map:
The electoral college count would have been 332 for the Democratic candidate, 206 for the Republican candidate. We can actually quantify how surprising the actual results were using the Kullback–Leibler divergence, but to make sense of this we should compare it to previous elections:
| Election Year | Surprise (in bits) |
|---|---|
| 1988 | 0.8504352 |
| 1992 | 33.3329646 |
| 1996 | 2.8056336 |
| 2000 | 2.8825127 |
| 2004 | 1.3989161 |
| 2008 | 0.7251642 |
| 2012 | 0.3414356 |
| 2016 | 3.5202876 |
For 1992, remember Ross Perot captured about 20% of the popular vote, the most a third party received since Teddy Roosevelt ran for a third term on the Progressive Party ticket in 1912,3Perot's 1992 performance stands third in the rankings of "percent of popular vote a third party candidate received in the presidential election". Teddy in 1912 stands at first with 27.39% of the vote, Millard Fillmore's 1856 bid on the Constitutional Party ticket ranks second at 21.54% of the vote, and Perot's 1992 bid comes in third at 18.91% of the popular vote. In contrast, Gary Johnson's 2016 bid received 3.28% of the popular vote. which is why it is the most surprising row in our table.
So what happened? Why was this so surprising? There are a variety of ways to approach answering this. We may take the expected vote proportions and compare them to the actual vote proportions, and extrapolate out the difference in votes (assuming voter turnout remained the same in this hypothetical 2016 election as in the actual 2016 election). We find the difference in votes:
| Party | Difference in votes | As Percent of Total Vote |
|---|---|---|
| Third Parties | 4,397,608 | 0.0321493 |
| Democratic | -2,605,576 | -0.0190484 |
| Republican | -1,792,032 | -0.0131009 |
Although we find Trump underperformed by 1.31%, we find that Clinton underperformed by 1.9% of the vote. Third parties overperformed by roughly 3.21% of the popular vote. Note: Johnson received 3.28% of the popular vote in 2016 — does this account for the overperformance of third party candidates in 2016?
Two questions immediately emerge: (a) which Third party candidate overperformed? (b) If we supposed the third parties received lower votes (i.e., they received the expected votes), how would the difference be re-allocated between Trump and Clinton?
Measuring Third Party Overpower
We can actually measure the Kullback–Leibler divergence for how Johnson performed in 2016 compared to 2012 (lumping all non-Johnson votes in one category). This measures the surprise in Johnson's 2016 performance relative to 2012 expectations, an adequate way to gauge improvement. We may do similarly for Stein, since both ran in 2012 and 2016. The result may be summed up thus:
The ratio of "Johnson's improvement" to "Stein's improvement" averaged 22.88781 — that's over an order of magnitude improvement!
In the states which went from Obama in 2012 to Trump in 2016 (specifically: Florida, Michigan, Pennsylvania, Wisconsin, Iowa, Indiana, North Carolina, Ohio, Nebraska), Johnson doubled his votes in 2016 compared to 2012...or better (in Florida, Johnson saw his votes grow from 44,726 to 207,043). Johnson's average improvement among these swing states was 336% more votes in 2016 compared to 2012. We should observe that Johnson didn't run in either Michigan or Wisconsin in 2012, though.
These numbers tell us, quite simply, Johnson improved considerably between 2012 to 2016. This alone is quite surprising, third party candidates seldom improve so drastically. Jill Stein, on the other hand, saw very little improvement in votes. We may safely conclude that Johnson is the dominant (sole?) dynamo for the third party's surprise improvement, which answers the first question the previous section posed: Which Third party candidate overperformed?
We may safely answer, it was Johnson.
A Wonderful Life
The Economist's Lexington, stating the obvious, informs us, Most of those who voted for Mr Johnson in 2016 were protesting against the alternatives.
But if there were a timeline where there was no Libertarian ticket in 2016, or any other third party for that matter, how would the election have changed?
Looking at the numbers, third party voters changed the outcome in Flordia, Michigan, Pennsylvania, and Wisconsin. (They did so in North Carolina, but they would have to break 95.7% for Clinton, which is implausible.) If 68.997%+1 of third party voters broke for Clinton in this hypothetical, then Clinton would have won an additional 75 delegates in the electoral college. This would have changed the outcome of the election. Is this feasible?
FiveThirtyEight's Harry Enten asked this very question in his article, Election Update: Is Gary Johnson Taking More Support From Clinton Or Trump? If we take his observations as a launching off point, then third party voters are divided up thus: 1.19864% (of the total vote) is taken from the third party candidates, then the remaining third party voters are divvied up evenly between Trump and Clinton. This effectively erases the margin of victory for Trump. (With the exception of Florida, only a small fraction of third party voters need to be shaved off to change the outcome.)
| State | Delegates | Trump margin | Shift to Clinton |
|---|---|---|---|
| Florida | 29 | 0.0119863 | 0.0207737 |
| Michigan | 16 | 0.0022303 | 0.0311395 |
| Pennsylvania | 20 | 0.0072427 | 0.0228425 |
| Wisconsin | 10 | 0.0076434 | 0.0366399 |
In this alternate timeline, where third parties vanished and its constituents had to pick between Trump and Clinton, would have produced a drastically different result.
Even if taking Harry Enten's findings too generously, that Clinton's edge was not 1% but a more conservative estimate 0.7643%+1 (the margin enough to win Wisconsin), in that hypothetical Clinton would still lose Florida but win Michigan, Pennsylvania, and Wisconsin. This would have given Clinton 46 electoral delegates, enough to make her delegate count 278 to Trump's 260. Again we find Johnson acted as spoiler, prevented Clinton's victory, and delivered to us a Trump presidency.
Remark. Let us suppose Enten's findings could be used to construct a random variable describing how third party voters will likely vote. Given that polls have a margin of error of 0.04 at a 95% confidence level, we can construct a normally distributed random variable X centered at 0.01 [which Enten determined is the edge Clinton has] with a 0.02 sigma [from the noise for polling] for the third party supporter who just "randomly" picks who to vote for as follows: generate a random real number following this distribution and, if it is positive, vote for Clinton, otherwise vote for Trump. In this scheme, Trump receives 30.85375% of the third party voters, Clinton receives 69.14625%, enough for Clinton to win Florida, Michigan, Wisconsin, and Pennsylvania.
Exercise 1. The New York Times's Libertarian Gary Johnson Polls at 10 Percent. Who Are His Supporters? surveys the demographics of Johnson supporters. Consider using a MRP model (like Gelman et al.'s in arXiv:1802.00842) to estimate the preference of third party voters.
Exercise 2. The argument produced above, and the results of exercise 1, give two different ways to show Johnson was a spoiler candidate and Clinton would have won the election had Johnson not run. But it is not wise to go to see with two chronometers (take either one or three). Think of another test for showing Johnson was a spoiler candidate.
What Remains to be Investigated?
Aside from the exercises for myself, points worth pursuing include what could Clinton have done differently? It's one thing for us to sit back and say, "Well, well, third parties ruined everything." But it's more useful to consider how third parties attracted voters, and what Clinton could have done to counter this effect.
Also comparing the demographics of Obama-Trump supporters to Johnson supporters may be insightful. If it turns out these two share a suitably similar political culture, then we may have found one strata of swing voters. It remains to be seen if they are so fed up with Trump that they abstain from even voting in 2020.
But also worth considering is the newly energized Democratic base which didn't materialize for Clinton in 2016 but sure as Hell materialized to protest Trump and vote out Republicans in 2018. If the newly energized base is larger than the Obama-Trump and Johnson voters, especially in the swing states, then it may be worth considering alternative 2020 strategies.
All the scratchwork for this post may be found on Github.
No comments:
Post a Comment