Knowledgeable individuals protect the wisdom of crowds

By Ed Yong | September 13, 2011 7:00 pm

If you ask someone to guess the number of sweets in a jar, the odds that they’ll land upon the right number are low – fairground raffles rely on that inaccuracy. But if you ask many people to take guesses, something odd happens. Even though their individual answers can be wildly off, the average of their varied guesses tends to be surprisingly accurate.

This phenomenon goes by many names – swam intelligence, wisdom of the crowd, vox populi, and more. Whatever it’s called, the principle is the same: a group of people can often arrive at more accurate answers and better decisions than individuals acting alone. There are many examples, from counting beans in a jar, to guessing the weight of an ox, to the Ask The Audience option in Who Wants to be a Millionaire?

But all of these examples are somewhat artificial, because they involve decisions that are made in a social vacuum. Indeed, James Surowiecki, author of The Wisdom of Crowds, argued that wise crowds are ones where “people’s opinions aren’t determined by the opinions of those around them.” That rarely happens. From votes in elections, to votes on social media sites, people see what others around them are doing or intend to do. We actively seek out what others are saying, and we have a natural tendency to emulate successful and prominent individuals. So what happens to the wisdom of the crowd when the crowd talks to one another?

Andrew King from the Royal Veterinary College found that it falls apart, but only in certain circumstances. At his university open day, he asked 82 people to guess the number of sweets in a jar. If they made their guesses without any extra information, the wisdom of the crowd prevailed. The crowd’s median guess was 751.* The actual number of sweets was… 752.

This collective accuracy collapsed if King told different groups of volunteers about what their peers had guessed. If they knew about the previous guess, a random earlier guess or the average of all the earlier guesses, they overestimated the number of sweets. Their median guesses ranged from 882 to 1109. King likens this effect to real-world situations where people collectively drive the prices of items above their value and create economic bubbles. It’s what happened to create the recent US/British housing market crash or, more historically, the tulip mania of 17th century Holland.

Jan Lorenz recently found the same thing. Swiss college students can form a wise crowd when answering questions independently, but once they could find out what their peers had guessed, their answers became more inaccurate. In his summary of the study, Jonah Lehrer wrote, “The range of guesses dramatically narrowed; people were mindlessly imitating each other. Instead of canceling out their errors, they ended up magnifying their biases, which is why each round led to worse guesses.”

Is the crowd doomed to groupthink? Not quite. King found that he could steer them back towards a wiser guess by giving them the current best guess. When this happened, the median returned to a respectable 795. So the crowd loses its wisdom when it gets random pieces of information about what its members think, but it regains its wisdom if it finds out what the most successful individual said.

King says that this mirrors what happens in real life. The crowd may be a social beast, but it isn’t an indiscriminate one. Certain individuals wield disproportionate influence, and groups of soldiers, employees, players and even animals often rely on leaders when they make decisions.

There’s a reason for this. When King provided his volunteers with the best previous guess, their range of answers was narrower with fewer extreme predictions. Their collective answers were also about as accurate in small groups of 10 people as they were in larger ones of 70. King writes, “Copying successful individuals can enable accuracy at both the individual and group level, even at small group sizes.”

But King’s study still reflects an artificial situation, because he knew beforehand what the right answer was and could provide the crowd with the closest guess. Real crowds rarely, if ever, have that luxury. If anything, this results simply reiterates how important it is to choose who we emulate. If we pick poorly (like the crowds who learned about a random earlier guess), our decisions are worse. If we pick well (like the ones who learned about the best previous guess), we fare better. You can insert your own modern case study here, but perhaps this study ends up being less about the wisdom of the crowd than a testament to the value of expertise. Maybe the real trick to exploiting the wisdom of the crowd is to recognise the most knowledgeable individuals within it.

* Yes, I’m using the median. The last time a science writer did this for a wisdom-of-crowds story, the internet erupted. For anyone not convinced by the median, this post by Josh Rosneau lays it all out clearly. Stats junkies can pore over the data for themselves in the image below.

Reference: King, Cheng, Starke & Myatt. 2011. Is the true ‘wisdom of the crowd’ to copy successful individuals? Biology Letters http://dx.doi.org/10.1098/rsbl.2011.0795

Image from despair.com

Comments (18)

  1. Todd Masco

    To me, the way information leaks into the system by providing the “best earlier guess” isn’t a minor flaw, it invalidates the whole thing. In the wild, there is no input of objective information (or at least, nothing that people can agree upon as an objective source: scientists vs. industrial lobbyists, political operatives vs. researchers, etc).

  2. Like what I said in the last paragraph, right? ;-)

  3. Amos Zeeberg (Discover Web Editor)

    It’d be more interesting [and probably more realistic] if he’d picked out “leaders” ahead of time and told subjects what the leaders had picked. Could’ve also analyzed the effect of different methods for choosing leaders: performance in previous estimation, self-identified skill at estimation, etc.

  4. Tim Williams

    @Amos, I don’t see a real difference between picking leaders at random, at the start, or picking leaders at random as the experiment progresses.

    @Ed, I agree with Todd; your last paragraph isn’t worded strongly enough. That said, if we assume that the best leader is as unknowable as is the best answer, and an individual chooses to follow a leader, they are only hurting their chances of getting the right answer. There is no given correct answer for who should win an election. I would like to see a study where people were asked to choose the most soothing color from 10000 paint swatches. There would be no objective answer that isn’t dependent on human preference. What would be the median if people were asked individually? How about if they knew the running mean? There is no obvious “greater” value for people to err on. If they knew a random previous answer? Would bimodality develop, like in the US two-party system?

  5. Paul

    I agree with the previous commenters. This is not a case of “emulating the most successful individual” it’s being given information from outside the experimental situation about how close the guesses are.

    This would actually be a valid experiment if they tested each participant several times and established who were the most accurate guessers on average, then on a SUBSEQUENT guessing task the crowd was given the names of these supposedly accurate guessers and their guesses on the current task.

  6. Andrew King

    Great article Ed – thanks for featuring the paper!

    It was a lot of fun to do the experiment, and I am glad it is attracting interest. I enjoyed reading all the comments!

    @ Todd – Yes, it isn’t perfect! But, regarding ‘objective information’ – several animal species have evolved specific signals that advertise the information that they possess, and expert leaders can emerge even when individuals do not know how the quality of their information compares with that of others see http://www.cell.com/current-biology/abstract/S0960-9822(09)01412-2. Also, in this task, everyone guessed sequentially, so I imagined it to represent people estimating the quality of a feature one after another (like habitat or mate choice, or foraging patch choice, like we discuss in the paper). But even then, identification of an expert may not always be easy, and may be prone to error, especially where group composition is unstable and opportunity for repeated interaction limited. Therefore I’m testing these ideas further insystems that I can track individual performance, and manipulate information under controlled settings (e.g. fish).

    @ Amos, @ Paul – I think Tim is correct. What do you predict the difference to be if you give someone the following information: “this person, X, has done well in previous guessing tasks, and has guessed N” versus “this person, X, is the best guesser in the crowd so far, has guessed N” (which is what people were given). I am not sure I expect there to be any difference. Also see this paper we wrote on the topic if you are interested http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0015505

    @ Tim (and Amos), yes, bimodality does help (at least in theory – we modelled it, link to paper above). For repeated decisions – where individuals are able to consider the success of previous decision outcomes – the collective’s aggregated information is almost always superior to that of the best performing individual.

    Cheers, Andy.

  7. Awesome. I love it when scientists field questions about their own research. Cheers Andy.

  8. JMW

    I think an interesting set of follow up experiments would revolve around how “leaders” are chosen.

    1) Re-run the experiment by giving each person the best previous guess;
    2) Re-run the experiemnt by asking each individual how certain they are of their answer, and give the next person the guess of the person who self-scores highest on the certainty scale.

    I doesn’t take a genius to see where I’m going with this…but it would be intersting see which of the two methods ends up with the more accurate median guess.

  9. Sergut

    If anyone is interested, James Surowiecki gives more examples of crowds not being “wise” due to communication among the individuals (leading to lack of “independence”) in his book “The Wisdom of Crowds”. He also provides several ideas about how to achieve independence in different context (and thus “wiser” crowds). Good reading.

  10. SRC

    Great article! I like the use of medians, but would like to point out that as with the mean, they should be accompanied by some measure of the variance (range, interquartile-range).

    It is interesting to point out that in “Best earlier guess known” scenario the distribution of guesses seems to approach normality: in the other cases, there are still a lot of outliers. Oddly, the only other situation which seems to tighten the distribution is “Random earlier guess” scenario.

    I wonder what happens when the largest guess is the only information available to study subjects: would this necessarily lead to wildly higher and higher guesses and lots of variability. How about in the opposite direction, the smallest guess? Can this help us understand how the crowd reacts to “extreme” leadership as opposed to “Random” or “Best” or “Average”.

  11. Pam Dunker

    @Tim – color preference is not as subjective as you seem to think. Various studies on color influences on people demonstrate trends that would not otherwise appear if true stochastic subjectivity were at work.

    @Andrew – it seems a lot like an experiment in feedback loops that are used to regulate or at least constrain outcomes. The trick here seems at least two-fold; 1-the input(s) are not known but rely on normalized variance which can be skewed by wildly variant factors; and as noted above 2- there is no way of determining the best or most accurate solution for most problems; but also for most problems the correlated outcome’s affect on the participants also to some extent re-calibrate the set points for how people choose in the next event’s poll.

  12. Steve

    I don’t understand why, when the obvious conclusion is that the best conclusion is reached with no “leaders” at all (i.e., only one off from the actual number), virtually all the comments are about the issue of leadership; why is there still a need for a “second best” methodology?

  13. vc

    interesting article, except…

    “people collectively drive the prices of items above their value and create economic bubbles. It’s what happened to create the recent US/British housing market crash ”

    lolwut

    http://mises.org/

    http://mises.org/journals/scholar/Thornton13.pdf

  14. Phil

    Very interesting article. I am a meteorologist and there has been research that has shown that a consensus forecast of many human meteorologists are generally superior to any individual forecast especially in the aggregate. However, this article has me wondering how the consensus forecasts were derived. When I was in college the individual forecast that created the consensus were done similar to the first experiment – little interaction between the people who make the forecasts. It wasn’t a bubble – information from others did leak in and out – but it was close. Where I work as a forecaster, a modified form of this type of consensus has been tried but in this case, the forecasts of other individuals are known and likely influence the decision. It would be an interesting experiment to have two groups of meteorologists make forecasts – one in a collaborative process and then take the consensus of their forecasts and one where they are individually making th forecast with little or no input from others. Prior to reading this article I would have said the former would give the best forecast but now I am not so sure.

  15. Mike Cope

    Using a system where each guesser was told the Best Guess So Far would home in on the correct figure, if a better guess replaced it, and should do so fairly quickly since each guesser in not estimating how many beans or sweets, but how close the ‘best’ might be to the actual figure. Later guessers need guess in a smaller and smaller range.

  16. JohnReddit

    Wtf is “swam intelligence”?

  17. sqze

    I like the paintchip idea and I do believe people will pick the same color as the leader, especially if they knew who the leader was and admired that person. If it was said that Angelina Jolie chose persimmon and Taylor Swift chose cotton candy pink then you might find there is a two party system developing.

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Not Exactly Rocket Science

Dive into the awe-inspiring, beautiful and quirky world of science news with award-winning writer Ed Yong. No previous experience required.
ADVERTISEMENT

See More

ADVERTISEMENT
Collapse bottom bar
+

Login to your Account

X
E-mail address:
Password:
Remember me
Forgot your password?
No problem. Click here to have it e-mailed to you.

Not Registered Yet?

Register now for FREE. Registration only takes a few minutes to complete. Register now »