These are my random notes on polling. I don't expect anything revolutionary to be contained here, I'm just hoping to consolidate them in one place.
Topics:
- Margin of error
- Likely voters
- Averaging polls
- Conducted by phone or by internet (think about, add later)
Margin of Error
Big idea. When we conduct a poll, we ask a subset of the population for their response. There is some error if we extrapolate the results from the poll conducted on this sample of the population, and try to apply it to the population as a whole. By "error" I mean "The estimates will be off 'by a few percentage points'." I do not mean the extrapolation is invalid.
The "off by a few percentage points" is called the sampling error. We can estimate it, and the estimate is referred to as the "margin of error."
Mathematical Details
Intuition: The margin of error for a poll of n respondents (out of a population of N individuals) asked a question is the width of the confidence interval of the response.
For a "large enough sample" on a binary question, we have a binomially distributed sample, and can use the normal approximation. We then determine some level of confidence \(\gamma\) to determine a z-score \(z_{\gamma}\) using the quantile function for the Normal distribution, which tells us how many standard deviations wide the confidence interval needs to be. We approximate the standard deviation using the "standard error", which in turn is approximated by \(\sqrt{s^{2}/n}\) the squareroot of the sample variance of the response divided by the sample size.
This is relatively unenlightening, there are technical matters which (I think) are contentious (at least, from a Bayesian perspective). It's also really hard to interpret the margin of error (it's easy to misinterpret it as "95% probability the true value lies in this interval", whereas it's really saying: "If we repeated this poll a large number of times, 95% of those polls would result in a confidence interval containing the population parameters").
Puzzle MOE1. Is there a better Bayesian replacement for the margin of error for a given poll? Presumably credibility intervals, but is there a quick way to get it without heavy computation?
Heuristic. The 95%-confidence margin of error for a binary question on a survey of n respondents may be approximated as \(1/\sqrt{n}\).
This is because the margin of error would be bounded (i.e., less than or equal to) the case where the true probability (proportion of "yes" responses) is 1/2, which produces \(moe = z_{0.95}\sqrt{0.5(1-0.5)/n} \approx 1.96\times 0.5/\sqrt{n}\leq 1/\sqrt{n}\).
Coincidentally, if we used Bayesian reasoning, and estimated the posterior distribution of the proportion of the population who would answer "yes" using a Beta distribution updated with the survey data, then the width of the 95% interval is also decently approximated by \(1/\sqrt{n}\). (Using \(2\sqrt{\operatorname{var}[\theta]}\) gives approximately the same result, but \(1/\sqrt{n}\) is for pessimists like me.)
Nonresponse error
One difficulty to note is if someone being polled by phone...hangs up before completing the survey. (Or, if in person, walks away from the questioner, or whatever.) If this happens sufficiently frequently, it impacts the reliability of the poll, and really increases the margin of error of the poll.
For many years, the response rate was viewed as a measure of the poll's quality. This heuristic is hard to validate.
We don't have an adequate way to digest polls with a high incompletion rate, or even what qualifies as a "high incompletion rate".
Puzzle MOE2. Can we have some approximate formula relating the nonresponse rate to the poll quality?
Likely Voters
Some polling firms ask questions to gauge if the respondent is a likely voter or not. What does this mean? Not every registered voter votes. We'd like to filter out the nonvoters. What's left are generically referred to as "likely voters". The exact statistical model sometimes remain undisclosed, it's the "secret sauce" for polling firms. (Gallup being a notable exception.)
There was some work done by Pew suggesting the likely voter model works fairly well, but can be improved if the respondent's voter history were known (and improved further with some magical machine learning algorithms).
The quality of a poll improves when it reports the results from likely voters, though this is far more costly to the polling firm.
Puzzle LV1. Is there some statistical way to infer how the reliability improves when a poll surveys likely voters as opposed to registered voters?
Poll Aggregation
This is the fancy term used for "combining polls". Let's consider some real data I just took from RealClearPolitics:
| Poll | Date | Sample | MOE | Biden | Trump | Margin |
|---|---|---|---|---|---|---|
| Economist/YouGov | 5/23 - 5/26 | 1157 RV | 3.4 | 45 | 42 | Biden +3 |
| FOX News | 5/17 - 5/20 | 1207 RV | 3.0 | 48 | 40 | Biden +8 |
| Rasmussen Reports | 5/18 - 5/19 | 1000 LV | 3.0 | 48 | 43 | Biden +5 |
| CNBC | 5/15 - 5/17 | 1424 LV | 2.6 | 48 | 45 | Biden +3 |
| Quinnipiac | 5/14 - 5/18 | 1323 RV | 2.7 | 50 | 39 | Biden +11 |
| The Hill/HarrisX | 5/13 - 5/14 | 950 RV | 3.2 | 42 | 41 | Biden +1 |
| Harvard-Harris | 5/13 - 5/14 | 1854 RV | 2.0 | 53 | 47 | Biden +6 |
There are a variety of ways to go about it. The most dangerous way is what RealClearPolitics does: just take the average of responses. For example, take the column of respondents favoring Biden, then take its average (which R tells me is 47.71429%). For Trump, the simple average is 42.42857%. Together, this sums to 90.14286% (only one poll, Harvard-Harris, has Biden and Trump sum to 100% support).
We don't have any way to gauge the margin of error of this estimate, though, and we don't reward larger polls anymore than smaller polls.
If we took the weighted mean (weighted by the sample size), Biden would receive 48.30791% and Trump 42.80864% with the weighted mean response at 91.11655% favoring one or the other.
We can further adjust weights, rewarding likely voter polls (or penalizing all others; for example, weigh registered voters proportional to the fraction of registered voters who turned out to vote in the last presidential election, something like 0.58).
The margin of error is all too frequently misinterpreted. It's probably better not to contrive some composite margin-of-error.
References
- Margin of Error
- Andrew Mercer, 5 key things to know about the margin of error in election polls. Pew Research, 8 Sept 2016.
- Gary Langer, Sampling Error: What it Means. ABC, 8 October 2008
- Likely voters
- Scott Clement, Why the ‘likely voter’ is the holy grail of polling. Washington Post, 7 Jan. 2016
- Gallup(?), What is the difference between registered voters and likely voters? Gallup, not dated.
- Scott Keeter and Ruth Igielnik, Can Likely Voter Models Be Improved?. Pew Research Institute, 7 January 2016
- Carl Bialik, Election Handicappers Are Using Risky Tool: Mixed Poll Averages. Wall Street Journal, 15 Feb. 2008
No comments:
Post a Comment