Bias in psychology

By Razib Khan | May 16, 2012 10:46 pm

Ed Yong has a piece in Nature on the problems of confirmation bias and replication in psychology. Yong notes that “It has become common practice, for example, to tweak experimental designs in ways that practically guarantee positive results.” The way this has been explained to me is that you perform an experiment, get a p-value of > 0.05 (significance). You know that your hunch is warranted, so just modulate the experiment, and hope that the p-value comes in at < 0.05, and you have publishable results! Obviously this is not just a problem in psychology; John Ioannidis has famously focused on medicine. But here’s a chart which shows that positive results are particular prevalent in psychology:

There are many angles to this story, but one which Ed did not touch upon is the political homogeneity of of psychology as a discipline. The vast majority of psychologists are political liberals. This issue of false positive results being ubiquitous is pretty well known within psychology, so I’m sure that that’s one reason Jonathan Haidt has emphasized the ideological blinders of scholars so much. Let’s assume that the range of false positives to support a wide array of hypotheses is rather large. In other words, if you have the will, you can support many alternative hypotheses. How then do you support your hypothesis? In all likelihood, consciously or unconsciously, you are guided by normative considerations. From the pot of “statistically significant” results you just peel away the ones which align with your preferences.

All of this is one reason why I’m rather skeptical whenever I hear that a psychologist has dispassionately waded into a domain of study and come back with objective and incontrovertible evidence supporting their own position. I can go in and do that too. Or more concrete, how hard has it been for your to find “sources” which support whichever crazy opinion you want to hold on Google?

Knowledge is hard.

MORE ABOUT: Psychology

Comments (26)

  1. mdb

    so I’m sure that that’s one reason Jonathan Haidt has emphasized the ideological blinkers of scholars so much.

    should be blinders

  2. marcel

    in re: ideological blinkers

    I like this metaphor, similar to fireflies signalling each other (for mating purposes I think). Or maybe like a turn signal – “I’m about to go left”. Then you get the old farts in the field who leave their blinker on forever but never actually follow up on what they are signalling.

  3. Ian

    @1 – blinkers is the correct English word, blinders the American version.

  4. Darkseid

    as a PSY major, I can assure you that most people in psychology won’t care about this and also won’t understand what this means. it’s not a “real” degree IMO

  5. April Brown

    @ #4, and ironically, this is exactly the sort of phonmenon you’d expect a psychologist to study. I guess you can’t expect a profession to point it’s own weaknesses. Bad for funding.

  6. Psych researchers care a great deal about the problems in the research (another issue whether something will be done).

    I think the ideological bias question has relevance to some areas of psychology, but not all. I don’t think political ideology would impact basic perception and memory research, or action research (although, of course, there are other preconceptions that may influence how you interpret your data). I see the problems of ideology in some parts of social psychology (which I deal with in part), and I think that is a problem.

    I’m very happy about Haidt bringing it to the forefront (and on occasion I do challenge people when there is this knee-jerk pathologization of what is to the right – because I think it fucks with results and interpretation, and that really matters more to me than ideology – then I’m not interested in politics. Of course, I’m a low-producing academic in the corner of scandinavia, so what Impact do I have).

    I think the problems that Ed brings up (and which are brought up by lots of researchers, even outside psychology) are broader than the ideology issue. Although it does have a place in the critique.

  7. As with most fields, psychology is probably divided between the “critical psychologists” and psychologists doing objective research. The chart above possibly demonstrates that there are far more of the former than the latter in the field. A main tenet in any “critical theory” in any human sciences discipline is that you only work with subjects whom you want to help, which means, of course, that you’re only going to posit “ethical” hypotheses about them and that you’re damned well going to support those hypotheses.

    From a textbook on qualitative research that I have: “We are distressed by underprivilege. We see gaps among the privileged—the patrons and managers and staff—and underprivileged participants and communities. We often devote some of the study to the issues of privilege, coming up with research questions that illuminate or possibly might alleviate underprivilege . . . We see democracies needing the exercise of public expression, dialogue, and collective action. Most case study researchers try to create reports that provide grounds for and stimulate action.”

    This same attitude is embedded in every “critical” approach to research, quantitative and qualitative. Advocacy is all fine and well, but it obviously taints research.

  8. AG

    Looks like hard science (physics, engineer, ect) is hard to manipulate.

  9. Objective science? Objective by whose ruler? Who is measuring the objectivity? How can we know? It is financially imperative that the hypothesis gets proved. I heard the same thing go on with biological research. How can we spin it, what results can we throw out? so it matches our conjecture.

    Science is not science… other wise we would celebrate the publishing of the NOT true or ambiguous as much at the tentative truth. When they only reward positive results, you end up with duplicate projects all searching for the same thing the same way.. to a dead end.

    When science has a place for.. “This did not work” publications.. THEN and only THEN will I believe it is trying to be objective.

  10. #9, well, to fix the problem you need to perhaps chill out a little 😉

  11. Sandgroper

    People do publish on engineering failures – not just theory that didn’t work out, but real live failures. Not as much as it would be good for the state of knowledge to do, but it happens.

    In engineering there is a very big disincentive to getting things too far wrong. In psychology, you can pave roads through hell with your crackpot theories (I stole that line from the dragon in Grendel). In psychiatry, the prognosis for serious mental illness is always so bad, that no one is going to blame you for not pulling off a miracle cure – which I have been told by senior people in the field is a big discouragement to going in for it – no matter how good you are, you’re not going to be able to save too many. Maybe you can prevent some from killing themselves, which I guess is a kind of victory.

  12. I don’t think the psych Ed is talking about is the “critical theory” psychology (Who does that? Oh, yeah, my old colleagues who are now replaced by the neuroscientists, and mathematical psychologist and an occasional emotin evolution social psychologist dabbler like me who also likes to remind the rest of the social faculty that, no, not all of us do critical theory. We, the behemoth over here who finance half of you do experiments).

    There is a problem even on the experimental (physics envy) side. There are extensive discussions about this on blogs, and within psychology (but the discussion also happens in other areas of science. I get a lot of discussions from the medical sciences, and also psychiatry which has a lot of issues right now with the DSM-5).

    Also, much of this is about the “not that kind of psychology” that I deal with. The area of us who do not want to help other people, but wants to look at psychological functioning.

    I get really curious about the engineering failure publishing… Of course, the “failures” in psychology is not anything spectacular. It is like the too many data-sets I have lingering where there are results, but, hell if I know what they mean, and thus, nobody will accept them for publishing. No good narrative to fit them.

    (Damn, maybe I should write this on my blog instead, I’m getting incoherent).

  13. if ‘critical theory’ was an explicit problem it would be less of an issue. no, the reality is a lot of psychologists think they’re being normative-free, when they aren’t. and as ase noted this is not as much of an issue in, for example, memory research. the structural problems/skew toward false positive bias and careerism in the academy though is a general dynamic, which instantiates in psychology in this particular manner.

  14. Sandgroper

    “I get really curious about the engineering failure publishing…”

    What are you curious about?

  15. What do you publish when you publish engineering failure? Because failure in Psychology is so often “failing to reject the null hypothesis”. (that in itself has problems) – so you simply don’t publish.

    So I’m kind of curious what this would mean. I started conjuring up that classic picture of the bridge that fell down due to resonance (tacoma bridge right?). Which would make sense. But, that is my imagination. So, what is published when publishing engineering failure?

    Engineering would in some ways also be different from basic research (where you don’t know what the heck will happen anyway).

    Maybe the simplest would be to give a short example.

  16. Now that I’ve read the article and not just Ed’s blog about it, clearly this isn’t an issue of critical theory run amok. (I was making assumptions based on the reference to Diedrik Stapel, whose fraudulent research was geared very much to the progressive, interventionist set.) Rather, it seems like a general antipathy toward falsification? Probably a problem in many disciplines. Many American linguists won’t go near a corpus because they know that, lurking within, are hundreds of sentences that, according to the linguists’ models of mental grammar, aren’t supposed to be there.

    Still, a paucity of attempts to replicate research can be affected by a dominant ideology within a discipline. If replication becomes a priority in psychology, it will be interesting to see which experiments are given intense scrutiny (through multiple attempts at falsification and complication) and which ones remain sacred cows that are not submitted to very much scrutiny.

  17. Chad

    To be honest I have never placed much trust in the “social sciences.” I recall far too much ideological bias in even the introductory courses I took as an undergrad.

    I am slightly disturbed by the potential biases indicated in the graph regarding Molecular biology, Genetics, Biochemistry, etc.

    But then I think this may be an inherent problem of these fields where the hypotheses tend to be “smaller.” In genetics, you may have one or two labs who work on a specific problem, so that the hypothesis is limited in its audience. Journals are not too impressed by negative results, so only positive results get published.

    Contrast this though to “Space Science” where a hypothesis may be more universal throughout the field. So negative results will have a wider audience and those have more appeal in publishing.

    At first I was a bit worried by the number of papers reported for each field by the chart, but after reading the methods and how they did randomize the selection, it seems reliable.

  18. Sandgroper

    In civil, structural and geotechnical engineering, we are designing for ultimate limit states, which are rare conditions we hope will never happen, and which are created in our imagination. When a limit state is reached and a structure fails, it is important to know why, and also how (because it makes a difference if something collapses catastrophically without prior warning or if it just starts showing some cracks and gives you years in which to fix it), because we learn far more from a failure than from the very many structures which don’t fail. It is very rare to build a full sized structure, and then load it to failure to see what happens, so we may not even know exactly how a structure will fail at an ultimate limit state, even if we can predict the limit state and its statistical probability of occurrence sufficiently accurately. This can be overcome to some extent by testing physical models in a centrifuge to simulate scaled-down gravity, but there are obviously limits to the complexity of the models that you can build inside a machine. People also do wind tunnel testing to simulate extreme wind effects, use flumes to simulate fluid flow problems, etc. There are many kinds of physical modelling, and also mathematical modelling. But nothing tells you as much as a real failure, provided the cause of failure is accurately identified, which can be difficult because often such events are the consequence of a complex series of phenomena acting in combination, or some unknown/unexpected physical property.

    The Tacoma Bridge is a good example. A more recent example is the WTC. Other examples are quick clay landslides in Scandinavia – sometimes the ‘failures’ are not failure of a structure so much as inability to observe or predict some natural phenomenon that has some capacity to impact on people. Those are ‘real’ failures that revealed some gap in knowledge, foresight or practice because of the complexity of the phenomena – they were sort of ‘beyond imagining’ until they really happened. But to identify the gap accurately so that it can be plugged, as it were, you need an investigation which gets at the truth of why the failure occured or was not prevented – so it is basically the investigation and its findings that are published initially, which obviously get very rigorously discussed. That is then followed by discussion of how the gap should be plugged, and writing that into the standards and codes of practice. Now, every bridge designer knows about resonance, and I imagine every structural engineer on earth now knows what happens if you put enough jet fuel into a steel framed structure. And sure as hell, every geotechnical engineer in Scandinavia knows about quick clay slides and why they happen.

    Mostly you are right, but some of this is in the realms of basic research, for example researching the properties and behaviour of naturally occuring earth materials like clays, weathered rocks and so on that do not behave like previously encountered and well-researched materials, or researching the occurrence and probability of extreme wind and wave conditions. You do get engineering researchers who publish research ‘failures’, in that materials or models were not found to behave the way they were supposed to in accordance with accepted theory – things can get a little heated when that happens, but it’s always important for those to be published and for that discussion to take place. Even if the researcher is saying ‘I found this, and I don’t know why it happened’, it’s important to say that, and sometimes engineering researchers do that – my experience of doing that at an international conference is that you suddenly have maybe a few hundred or a couple of thousand fellow engineers from all over the world all on your side trying to help you to figure it out.

    Sorry, that wasn’t really short.

  19. BDoyle

    You brought up statistics right off in the first paragraph. The bottom half of that chart is dominated by fields that rely heavily on controlled experiments with results validated statistically. Your ability to do controlled experiments in astronomy and geology is somewhat limited. Just an observation.

  20. Sandgroper

    In geology, the equivalent is to try to map, sample and test the natural variation. You’re right, it’s a difficult problem; it’s a lot easier with man-made materials. It’s made more difficult in that the way a geological material behaves in say a hand-sized specimen is going to be different from the way a mass behaves in the field.

    That pertains pretty much to all natural phenomena that I can think of.

  21. I would say that meditation research is far more prone to bias than general psychology is. There are committed skeptics who will ignore practicalities when informing everyone that “all meditation research sucks,” there are committed generalists who ignore any research that detracts from their conclusion that all meditations work the same, and there are True Believers, who ignore any research that doesn’t support their own pet meditation or meditation tradition.

    And it is not immediately obvious who falls into which category. Simply looking at the name of the institute or school that researchers work at can be very deceptive.

  22. Darkseid

    i had this post on reddit and it was doing pretty well….then i was notified it had been deleted by the mods in the science section due to “biased” headline. i titled it as “Psychology is “most biased” science. isn’t that the gist of what the paper was about? QED, i guess. i can almost guarantee you some PSY major complained.

  23. Sandgroper

    saijanai – it’s the nature of the beast. So far as I can gather, everyone who practices meditation gets something different from it, and everyone thinks the tradition they practise is the ‘right’ one. The advice I got about it turned out to be pretty good – keep it simple, don’t go into it expecting anything at all, just start practising it, stick with it and see what happens in your own particular case. Any general claims should be regarded with suspicion, and any individual stories are kind of irrelevant to your own case. Which makes it kind of difficult to persuade anyone to start doing it. There are even some reports of bad outcomes, I understand.

    Darkseid – engineers have a joke about geologists. The engineer is getting fed up with the geologist not being willing to commit himself to an opinion or predict anything. One day they are out driving and he sees a cow in a field. He thinks “I’ll get him.” So he says to the geologist “What colour would you say that cow is?” The geologist looks at the cow and says “Black and white…” “Ah” thinks the engineer, “Great – I got him to commit an opinion.” Then the geologist continues “…on THIS side.”

  24. Darkseid

    Sandgroper- oh man, that pretty much sums up a lot of people in general too. Lol, tho…i just can’t be stiffled by people like that so i try to avoid them.

  25. I wanted to respond something to Sandgroper (thanks BTW for responding to my question). But, it became long, so I did what Razib has asked at some point, and blogged it instead. Here it is, for what it is worth.

  26. Sandgroper

    Got it, Åse – thanks.


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Gene Expression

This blog is about evolution, genetics, genomics and their interstices. Please beware that comments are aggressively moderated. Uncivil or churlish comments will likely get you banned immediately, so make any contribution count!

About Razib Khan

I have degrees in biology and biochemistry, a passion for genetics, history, and philosophy, and shrimp is my favorite food. In relation to nationality I'm a American Northwesterner, in politics I'm a reactionary, and as for religion I have none (I'm an atheist). If you want to know more, see the links at


See More


RSS Razib’s Pinboard

Edifying books

Collapse bottom bar