Prediction is very hard

By Razib Khan | October 28, 2011 1:17 am

Nate Silver has an important post, Herman Cain and the Hubris of Experts. It’s not really about Herman Cain. Rather, it’s about the reality that pundits tend to underestimate uncertainty and complexity. Saying you don’t know isn’t as satisfying as making a definitive categorical assertion. This manifests particularly in the domains of sports and politics because there are clear and distinct criteria to assess predictive power. Politicians win or lose elections, while teams win or lose games. And yet despite the long history of minimal value-add on the part of pundits they persist in both domains. Why? I think it’s pretty obviously a cognitive bias toward storytelling. Similarly, in the 1930s the Alfred Cowles concluded that financial newsletters didn’t help their readers “beat the market,” but he also assumed these newsletters would persist. There was a psychological need for them.

The key here is to change the attitude of the pundit class. The populace will always have a preference for stories with plausible and clean conclusions over radical uncertainty. Not surprisingly many professional pundits reacted with hostility to Silver’s observation that they’re quite often wrong. I don’t venture into political punditry often, but when the Democrats passed health care reform I predicted that Mitt Romney would have no shot to win the the Republican nomination. The facts in this case seemed so clear. Romney was going to be walloped over and over again over his record on health care reform when he was governor. I was wrong. Romney may not win, but obviously he’s a contender. My logic was simple and crisp, but the logic was wrong. That’s why you let reality play out. If what was “on paper” determined national elections, then we’d be talking about President Hillary Clinton.

Of course political journalists that engage in analysis still have a role to play. Don’t newspapers have horoscopes and style sections?

MORE ABOUT: Epistemology

Comments (17)

  1. Pundits are entertainers – and occasionally thought provokers – but they aren’t seers.

  2. Don

    Rhazib: Direct hit. Don

  3. Two things that seem to help calibrating uncertainty: Prediction markets like If you are losing money you probably aren’t predicting things well. Also, it means that money gets to flow essentially from the bad predictors to the good predictors and lets everyone else have a nice market to automatically collate data.

    If one doesn’t want to spend money on this sort of thing, a free alternative is which just has you put down your estimates for how likely an event is, and let’s people put up different estimates. That website is still fairly new but is overall quite interesting. A lot of people have put down predictions on all sorts of things with ranges to the very short and personal (what will they have for dinner, whether they will go the gym that week) to more standard predictions about political elections, to very long-term predictions about the future of humanity. It does have an unfortunately addictive element.

  4. It’s funny you mention the difficulties with prediction… I did the very same thing last night, but in relation to Nicholas Ostler: Tea Leaves and Lingua Francas: Why the future is not easy to predict

  5. Clark

    The biggest problem with pundits is that all the incentives for their behavior are for creating controversy and thus viewership and discussion. There are almost no incentives for accuracy. Even were the future easier to predict we’d not expect pundits to be the ones making the correct predictions simply due to the incentive structures they are embodied within.

  6. Anon

    “My logic was simple and crisp, but the logic was wrong.”

    Never expect the world or its people to behave in a logical fashion. File this away under “lessons learned”.

  7. Chris T

    People naturally dislike uncertainty and disorder. Storytellers (pundits) provide a coherent narrative linking unrelated or random events to each other. Whether they are right or wrong is irrelevant, what is important is the sense of order and control they provide.

    Conspiracy theorists are a more extreme form of this phenomena.

  8. jb

    It’s been a long time since I have read pundits as telling me what is going to happen. Even when their articles are phrased that way, even when what they are literally saying is that something definitely is going to happen, I always read it as a vague suggestion of something that conceivably could happen, and I try to evaluate it against all the other suggestions of other things that could happen. Seriously, given the hundreds and hundreds of articles that a typical pundit writes over the course of a career, how often do you see them suffer any bad consequences for being wrong? It does happen occasionally, but really, not very often.

  9. +1 for making a prediction
    +1 for acknowledging a failed prediction

  10. just to be clear on something: i’ve long known most political predictions were stupid because of high uncertainty. i wagered the romney prediction because it seemed so airtight.

    1) romney presaged obamacare

    2) republicans hate obamacare


    but that’s not how it worked out. he could still lose. i think it’s 50/50. but there’s no way that i would have guessed that his position would be this strong this late. it brings home to me the reality of radical uncertainty in our understanding of large scale human psychology.

  11. JonF

    There’s a Jonah Lehrer piece from a couple of years back that I’m sure you’re aware of (called “Expertise”) in which he outlined a study wherein political pundits were wrong more often than chance. I’m not entirely sure if this is applicable here but I think it’s worth noting that it seems like a story to tell – any story to tell – is likely to be received better than post hoc analysis. Another cognitive bias I think that’s important here is telescoping. We tend to hone in on the one correct prediction made years ago and not on the litany of poor ones made in the mean time. I could have predicted every year in March over the past 90 years that the Red Sox would win the World Series. Most times I’d be wrong but in 2004 I was a genius.

  12. gcochran

    Pundits are more like cheerleaders. Ugly ones.

  13. Audiences aren’t interested in predictions that are likely to be true, such as when the sun will come up tomorrow, or that test scores in Compton will likely be lower next year than in Beverly Hills. Audiences like predictions in market competitions that are contrived to be close to 50-50: who will win Game 7 of the World Series or will the stock market go up or down next week.

    I have a long track record of making accurate predictions about things that most people find depressing or boring. There’s not much money in it.

  14. To make money with a prediction one must also be taking a contrarian position or there will be no betting action against you.

    Another fundamental problem with predictions is that any model you build against a set of historical data points will be only one of an infinite number other possible models that also explain the history equally well.

    David Hume even maintains that you can’t really know that the sun will come up at all until it actually does.

  15. Hermenauta

    “My logic was simple and crisp, but the logic was wrong.”

    Razib, maybe your only error was to overestimate the GOP´s rationality.

    Anyway, I think pundits are out there not to predict, but actually to hit the golden pot of self-fulfilling prophecy…

  16. “David Hume even maintains that you can’t really know that the sun will come up at all until it actually does.”

    But that’s the way to bet.

  17. Clark

    I thought Romney a shoe-in myself even though I don’t care for him. (I like Huntsman) He looks horrible until you start looking at the alternatives. Unfortunately the anti-Bush backlash that swept Obama in tended to clear out most of the experienced field of Republicans. The leaders that could potentially get people excited tended to have little experience (Rubio, Cristie) or else had big flaws (Perry, Bachmann, Pawlenty, etc.) That only left Romney. No one is excited by him but no one can find someone better.


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Gene Expression

This blog is about evolution, genetics, genomics and their interstices. Please beware that comments are aggressively moderated. Uncivil or churlish comments will likely get you banned immediately, so make any contribution count!

About Razib Khan

I have degrees in biology and biochemistry, a passion for genetics, history, and philosophy, and shrimp is my favorite food. In relation to nationality I'm a American Northwesterner, in politics I'm a reactionary, and as for religion I have none (I'm an atheist). If you want to know more, see the links at


See More


RSS Razib’s Pinboard

Edifying books

Collapse bottom bar