The crowds knows better than you?

By Razib Khan | November 21, 2012 5:23 am

Justin Wolfers & Betsy Stevenson have a piece up in Bloomberg, Crowds Are This Election’s Real Winners. In The Signal and the Noise Nate Silver has a chapter on Wolfers’ belief that prediction markets are superior to the sort of quantitative analysis that is his stock & trade. The belief isn’t based on an intuition. One of Wolfers’ graduate students produced a paper showing that Intrade was actually better than FiveThirtyEight in 2008. Silver demurs in the chapter because he suggests that the model which Wolfers and his student lay out in the paper had some modifications which allowed for one to judge Intrade’s performance superior. I’m willing to accept Silver’s assertion here because I’ve seen enough economic models which are modulated just enough to produce elegant and clean results. That being said, the general tone of the chapter is such that in his heart Silver seems to agree that Wolfers is fundamentally correct in the long term. Prediction markets, when done right, are more powerful than any analytic an individual could cook up.

All that being said, the economist’ faith on the power of mass market signals (“the crowd”) often strikes natural scientists as peculiar. When talking about elections it does seem that the “crowds” are going to be superior to the judgement of individuals or powerful quantitative models (after all, elections are about crowds!).  But there is a long history of the crowd being wrong in the very specific areas of natural science which rely on contingent and formal fameworks to make non-obvious predictions on somewhat complex systems. But that’s because in some areas of the natural sciences humans have a systematic bias due to intuitive psychological tendencies. Aristotle’s model was just more intuitively plausible than those of his skeptics’ for a few thousand years. And quantum theory would never win a crowd-vote. One Bohr is worth a thousand other humans. I think this long history of the worthlessness of mass market intuition across large swaths of the territory of science is why many scientists find technocratic solutions very appealing. The formal reflections of the elect has worked miracles in physics, so why not “social physics” (i.e., economics)?

Obviously there’s a difference between the expertise of a nuclear physicist, an economist, a financial entertainer, and an astrologer. When a physicist speaks about physics, you listen, for they are describing the world. When an economist speaks about economics, you listen, because they are reflecting honestly what economists think about the world. When a financial entertainer speaks, you listen for laughs, because they are mixing substance and style to entertain you. When an astrologer speaks you are a fool to listen, because they are using the artifice of science to sell you nonsense.

Epistemology is hard. There is no “Swiss Army knife” which allows one to know how to know. In some circumstances the utilization of statistics is a matter of style, to firm up a flimsy supposition with the rigorous garb of quantity. The delusion of false precision. But in other domains statistical knowledge is highly informative. And there are areas where one can usefully deploy deterministic models.

The problem that crops up is when one swims in one lagoon of the intellectual pool where a particular suite of tools is useful, it is easy to forget that the utility of that kit may be conditional on the characteristics of that domain. Many physical scientists seem to have a tendency of assuming that there can be a simple and elegant technocratic solution (I believe that this is one reason engineers are attracted to religious fundamentalism). Meanwhile, I have observed before that biologists are often totally wedded to the Malthusian model for humans, when that model has not been fruitful for nearly a century. Biologists may be correct over the long term, but as an economist once observed, over the long term we’re all dead. Finally, economists can see markets everywhere, when sometimes it is not supply & demand, but ecology which is speaking. In A Farewell to Alms I note that Greg Clark argued that the emergence of lactase persistence was a sign of the longstanding high per capita income of Northern Europe, because Northern Europeans could afford to consume large quantities of milk. But of course there are ecological parameters which are relevant for how rational it is to engage in dairy-culture.

In the end, the only solution I offer is trial & error iteration. There is no market, no decision tree, which can guide us here. What works, works. What does not, does not. We muddle on.

MORE ABOUT: Philosophy
  • dave chamberlin

    Vegas bookmakers have long known this to be true. They put out an early line on an upcoming game which is what the in house experts think it should be. But then as the bets come in the line moves in whatever direction the betting pushes it. The closing line is significantly more accurate than the opening line. It works because the really sharp betters, the consistant winners lay down huge bets while the lousy betters are always broke and make small wagers.

  • Mike Dickinson

    The crowd can be very good at predicting what the crowd will do. This is the basis of successful prediction markets.

    The crowd is MUCH less useful in determining things that the crowd as whole has no impact on, such as physical reality in the form of weather patterns, evaluating scientific theories, and many other such things.

    It can be a tool like any other, but the key is always knowing when it is the right tool for the job.

  • gwern

    Silver beat Intrade this year too, by the way: /

    On the other hand, I didn’t bother to do the Wolfers correction…

  • marcel

    In some circumstances the utilization of statistics is a matter of style, to firm up a flimsy supposition with the rigorous garb of quantity.

    Too often in economics (perhaps social science more broadly), the role of statistics resembles that of cavalry in modern warfare: to give tone to what otherwise would be a vulgar brawl.

  • Brian Schmidt

    On the crowds v. experts debate, my understanding is the crowd proponents concede that crowds are useless if they have no relevant knowledge or expertise, but that even a little expertise or information makes an individual into a useful participant. They also argue that crowds of experts are better than the median expert and often even better than the best expert.

    Silver’s averaging of polls seems like a partly crowd-based analysis, although he then adds his own framework. I don’t see why crowd analysis can only be done through prediction markets.

  • Matt

    Gwern, thanks for that post. With all this discussion on the performance of statisical models vs predictions, some actual numbers were highly welcome. I actually was just about to link your post in the comments when I saw you had.

    As for InTrade/prediction markets versus Nate Silver, I think InTrade really isn’t a great example in many cases. The main presidential elections were heavily betted on, but on most other propositions, there just wasn’t nearly as much data.

  • ryan

    #1, the assertion that the final Vegas line is more accurate than the initial line is fraught. You didn’t point to any source, so I’m wondering if you know this or merely assert it. It certainly ought to be possible to make more accurate predictions closer to a game since you start to see which injured players will return and which will not, so I’m not even clear on the usefulness of knowing that the final line is more accurate. I also have no sense of what the money looks like. All I know is that Vegas moves the line to equalize the payout with either outcome. Is this really crowd-sourcing, or are there big players that come in late and large when the crowd has bet poorly?

  • Arthur

    Note that economists only think crowds can predict well when the crowds have skin in the game.

    I think most economists believe even in other fields a prediction market would predict very well. Only people who knew the subject would bet. Mostly the scientist working on the field. But even the other betters would probably be following pretty closely the work on the field.

    Although I have doubts about bias in all kinds prediction markets, the logic of prediction markets working on science seems consistent with the logic of prediction markets working on economic things and elections.

  • jimbo

    Scientific American in a recent issue discusses quantum physics and irrationality, as related to humans in an election. Seems to me if irrationality due to quantum physics applies to humans in election scenarios then it will apply across most human endeavours.

  • http://www.isteve.blogspot Steve Sailer

    Prediction markets that reflect insider trading have an advantage over individuals lacking inside information. For example, the non-pecuniary Hollywood Stock Exchange encourages insider trading on movie box office numbers (e.g., by insiders who have seen studios’ marketing research).

  • dave chamberlin

    7) I am sorry I cannot cite a source but it is common knowledge amoung sports bettors that the line moves from an opening line to a final line that averages out to be a more accurate prediction. The final lines are sometimes influenced by either injury reports or weather and are therefore more accurate for these reasons. But for the large majority of the games there is no obvious reason for the final line to consistantly average better accuracy than the opening line which is set by very well paid Nate Silver type people. But it does confirming what the title of this post contends, crowds where one vote equals one dollar wagered are very shrewd indeed. The sharpest bettors have the consistant ability to place their bets early before the bets move in the direction of their bet giving them added advantage.


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Gene Expression

This blog is about evolution, genetics, genomics and their interstices. Please beware that comments are aggressively moderated. Uncivil or churlish comments will likely get you banned immediately, so make any contribution count!

About Razib Khan

I have degrees in biology and biochemistry, a passion for genetics, history, and philosophy, and shrimp is my favorite food. In relation to nationality I'm a American Northwesterner, in politics I'm a reactionary, and as for religion I have none (I'm an atheist). If you want to know more, see the links at


See More


RSS Razib’s Pinboard

Edifying books

Collapse bottom bar