Well, at least not the sort of metaphysical problems that Vox author Matthew Yglesias suggests are a problem.
“Except there’s a huge problem — we’re never going to know which model is correct.
…To anyone who understands probabilities, of course, this is nonsense….If you sit down at the blackjack table and play for a while, you will probably lose money. But you might not. Even the Washington Post’s current forecast that the GOP has a 95 percent chance of obtaining a Senate majority won’t genuinely be debunked by a Democratic hold. Five percent is unlikely, but unlikely things happen….
…But in an epistemological sense, the way we check probabilistic statements is to run the experiment over and over again. Flipping a coin twice doesn’t really prove anything. But if you flip it ten or twenty or a thousand times you’ll see that “it comes up heads half the time” is a good forecasting principle…
…we’re just never going to get the kind of sample sizes that would let us tell whose method of calculation is best.“
Cutting out all the background, that’s the heart of Yglesias’ argument (emphasis added). I’ll start by addressing the “problem” that we’ll never know which model is correct, we do have an answer to that.
“All models are wrong; some are useful.” -George E. P. Box
The correct question to ask is not which model is correct, but which model is more useful. Whether a given model is useful is highly subjective, to say the least. Even when we know that a model is deeply flawed it may still be considered to be useful. Take the Black-Scholes option pricing model, for example. We know that the Black-Scholes model has significant problems all the way down to the underlying assumptions not matching reality, but it’s still widely used for pricing options1. Why? Because it’s good enough for most investors and the results are known to be close enough to reality that it can be said to provide a useful result, even though it is known in advance that the result is wrong.
Now Yglesias is correct that observing an unlikely outcome does not, in itself, prove that a model is worse than another model that happened to predict the correct result this time. Yes, unlikely things do happen in the real world2, but why are you assuming that the assumptions that went into constructing the model are realistic3?
The assumptions underlying any model are simultaneously the strength and weakness of a model. We use models because we accept that the real world is too complicated to allow us to accommodate every single aspect of the system being modeled. The election forecasts use a poll of a small subset of the voting population to attempt to make predictions about the election in the future. There are two sources of possible error, first, the election happens at some point in the future and events can, and do, occur that can cause a significant number of people to change who they decide to vote for4.
The other possible source of error is that you are only polling a subset of voters and you don’t know whether or not they are representative of the entire population. If you had unlimited resources, you could in theory poll every single voter and likely achieve much greater accuracy (barring unforeseen events between your poll and the election). Needless to say, that’s not practical because that would amount to holding a poll that was effectively an election. Expensive and pointless.
I don’t follow election forecasts so I can’t say what exactly they do to attempt to improve the accuracy of the models. I can say that it is likely easy to find problems with the underlying assumptions of any poll that is 95% sure of the outcome. So the model can be debunked without needing to worry about the epistemological nature of probability. Now, given that such biased polls are put forward by the likes of The Washington Post, I’d still say an argument could be made that the model is useful, even if it’s stupid. After all, it’s making them money, isn’t it?
On a psychological level, most people are interpreting the forecast probability incorrectly. It doesn’t say that candidate X has a 60% chance of winning the election. It should be read as saying: candidate X would have a 60% chance of winning in the hypothetical universe of the model based on our observations of the real world and subject to the assumptions of the model. It’s telling IF the underlying assumptions hold that a particular outcome as the given chance of occurring.
So what does this mean as far as how you should view poll-based election forecasts? Honestly, I’d say you should always avoid using any model where you don’t understand the underlying assumptions and the model’s construction. You also need to know where the data used to fit the model parameters came from because that’s another possible source of bias. If you don’t know that much about the model you have no way to interpret what it is telling you, except to trust what others are saying that the model says. Your level of trust should be 0 when dealing with…really anyone who has either a financial interest in the model, or an ideological commitment to a particular result.
Really, if they don’t have a very long answer to the question, what’s wrong with this model, then you shouldn’t trust them.
1. Yes, I know the binomial model is more commonly used than Black-Scholes. The underlying assumptions are effectively the same between the two models and, for European style options at least, the binomial model will converge to the Black-Scholes model as the number of steps grows.
2. This glances over the question of how unlikely something has to be for it to be considered effectively impossible. Like most subjective things, going with your gut is not a good way to answer this question. A royal flush in poker is indeed unlikely, but it's not so unlikely as to have never happened in history. Contrast with a perfect bridge deal (assuming a fair deck) which has a probability of about . As my stats professor put it years ago, "if everyone who every existed played bridge continuously, the probability of ever seeing a perfect deal is still much less than one millionth of a percent. I'll leave it as an exercise for the reader to get a more specific result.
3. Yes, realistic is a rather soft term, but it's accurate. What's considered realistic has, to my surprise, turned out to be extremely subjective. Of course, from my point of view, I'd say that many people's idea of realistic has nothing to do with the real world.
4. There's a difference between uncertainty that can be measured as probability and actual uncertainty. That is, the risk that things we cannot anticipate will occur. You can never entirely eliminate uncertainty, but we try anyway. A great example of this was a psychology experiment I read about a long time ago. There are two urns, A and B filled with colored balls (red and blue). You are first instructed to pick a color, it doesn't matter if you pick red or blue. You next need to pick an urn, you win a prize (say, money) if the ball you pull out is the color you pick. You're told that urn A has 50 red and 50 blue balls in it. You are told nothing about urn B other than it has 100 balls in it. What urn do you pick.
A majority of people selected urn A, even though there's no advantage to doing so. Mathematically speaking you cannot make an optimal choice because you have no information about the distribution of balls in urn B. We pick urn A because we at least know the odds, even though it doesn't help us to know the odds. We hate uncertainty, something to keep in mind when thinking about probability and more generally when thinking about forecasting.