Which results from cognitive psychology are robust & real?

By Razib Khan | October 30, 2012 9:11 pm

A paper on the psychology of religious belief, Paranormal and Religious Believers Are More Prone to Illusory Face Perception than Skeptics and Non-believers, came onto my radar recently. I used to talk a lot about the theory of religious cognitive psychology years ago, but the interest kind of faded when it seemed that empirical results were relatively thin in relation to the system building (Ara Norenzayan’s work being an exception to this generality). The theory is rather straightforward: religious belief is a naturally evoked consequence of the general architecture of our minds. For example, gods are simply extensions of persons, and make natural sense in light of our tendency to anthromorphize the world around us (this may have had evolutionary benefit, in that false positives for detection of other agents was far less costly than false negatives; think an ambush by a rival clan).*


But enough theory. Are religious people cognitively different from those who are atheists? I suspect so. I speak as someone who never ever really believed in God, despite being inculcated in religious ideas from childhood. By the time I was seven years of age I realized that I was an atheist, and that my prior “beliefs” about God were basically analogous to Spinozan Deism. I had simply never believed in a personal God, but for many of earliest years it was less a matter of disbelief, than that did not even comprehend or cogently in my mind elaborate the idea of this entity, which others took for granted as self-evidently obvious. From talking to many other atheists I have come to the conclusion that Atheism is a mental deviance. This does not mean that mental peculiarities are necessary or sufficient for atheism, but they increase the odds.

And yet after reading the above paper my confidence in that theory is reduced. The authors used ~50 individuals, and attempted to correct demographic confounds. Additionally, the results were statistically significant. But to me the above theory should make powerful predictions in terms of effect size. The differences between non-believers, the religious, and those who accepted the paranormal, were just not striking enough for me.

Because of theoretical commitments my prejudiced impulse was to accept these findings. But looking deeply within they just aren’t persuasive in light of my prior expectations. This a fundamental problem in much of social science. Statistical significance is powerful when you have a preference for the hypothesis forwarded. In contrast, the knives of skepticism come out when research is published which goes against your preconceptions.

So a question for psychologists: which results are robust and real, to the point where you would be willing to make a serious monetary bet on it being the orthodoxy in 10 years? My primary interest is cognitive psychology, but I am curious about other fields too.

* In Gods We Trust and Religion Explained are good introductions to this area of research.

  • http://www.computerfiltermonitor.com alizardx

    Some studies on agnosticism vs religious believers might be illuminating. I see as much “will to believe” and “I have THE TRUTH” in many atheists as in True Believers in identified religions. A person who says “I don’t have the evidence” might be different.

  • https://plus.google.com/109962494182694679780/posts Razib Khan

    #1, mebee. technically though agnosticism is more a comment on the limits of knowledge than on your belief in a proposition.

  • John Ray

    I have had 200+ papers published on psych & related disciplines and my final conclusion is that it is all BS except for IQ

  • RedZenGenoist

    #2, that’s true, but most agnostics really don’t know what it means (or rather, don’t grasp agnosticisms comparative relationship to atheism)

  • RedZenGenoist

    #2, that’s true, but most self-styled agnostics really don’t know what the word means (or rather, don’t grasp agnosticisms comparative relationship to atheism)

  • CouriousScientist

    First, I think most cognitive psychologist would prefer to be categorized as being in the natural sciences as opposed to being in the social sciences (that might well be different for some social psychologists).

    Second, the approach to take a random paper on an esoteric/non-mainstream topic from a not really flag-ship journal to start the discussion about the validity of cognitive psychology is very questionable, to say the least. It is easy to find weird stuff in any discipline, the natural sciences included. For instance, if you want to make the ability to weed out all nonsense the criterion upon which to take a field serious or not, you could throw out theoretical physics (http://en.wikipedia.org/wiki/Bogdanov_Affair), and we surely don’t want to that.
    [Suggestion to commenter #3: If you really want to learn about psychology, start with reviews in Annual Reviews of Psychology, or some papers in Trends in Cognitive Science. If you’re hungry for more detail, continue with Psychological Review, Psychological Bulletin, JEP:G, JEP:LMC. If you don’t know the last 2 abbreviations, you should rather abstain from any judgement about the field of psychology].

    Now to some robust effects:
    – A fun one is the HINDSIGHT BIAS: The idea is that when people remember things that were a long time ago, they don’t really remember so much as they reconstruct. That is, they will respond what now they think they would have said. An example: If I asked you 3 weeks ago what the capital of Norway is, and you responded “Stockholm”, but in the mean time you learned that “Oslo” is the correct answer, and I now ask you what was your answer to my question about the capital of Norway three weeks ago, you would probably respond “Oslo”. Why would this be important? Well, because it tells us that memories like the one Razib describes about the musings of his 7 year old self about religion are not very reliable.
    – Another very robust effect is that we store verbal information (read or heard) in its auditory form in the short term memory (STM), hence the term “phonological loop”. When we access information in the STM, it is typically easiest to recall the first and the last items. The capacity of STM is 4 items for visual short term memory and up to ca 7 for verbal sort term memory.
    – Using the method of loci can help you to dramatically increase long term memory.
    – In the field of decision making it is clear that people are (on average) both risk and loss averse, even if there is quite some disagreement about the underlying cognitive mechanisms.
    – Also in the field of decision making it is clear the there is a trade-off between the speed and accuracy with which we can make most decisions.
    – Moving to social psychology / behavioral economics it is clear that on average people are not only motivated by monetary self interest, but find that equity and reciprocity also have some utility.

    There is caveat that one should keep in mind about these results, which is that the populations from which participants were sampled was in most cases WEIRD: (http://www2.psych.ubc.ca/~henrich/pdfs/WEIRD%20in%20Science.pdf, http://www2.psych.ubc.ca/~henrich/pdfs/WeirdPeople.pdf?). But htis does not mean the the effects are not robust! It means that we have to be careful with generalizations.

    I could go on for a long time (I just note that Psychological Bulletin often publishes meta-analysis, which are designed to identify robust effects), but that is not the point. The point is, that there are lots of very robust results (cognitive) psychology. But, as in most other sciences that examine complex phenomena (think climate science), variability in data leeds to publishing of non-existant effects (happens sometimes) and to disagreement about how the robust effects can be explained (happens all the times).

    Finally, I’ll point you to one part of cognitive psychological research where recent research showed that effects are not robust at all: It’s the association of intelligence and SNPs, where it turns out that a large scale study could not replicate any of the previously reported results (http://pss.sagepub.com/content/early/2012/09/23/0956797611435528). This is not to say that there are no SNPs effects, but it suggests that these are so small that even sample sizes of 1000 people are to small to detect them reliably (see here for the background: http://www.stat.columbia.edu/~gelman/research/published/power4r.pdf). [Note that I make no statement about genome wide associations studies, about which I know next to nothing].

  • http://evolvingthoughts.net John S. Wilkins

    Variance, not deviance. Human cognition is stereotypically teleological (lot of support for that) and this leads to the hyperactive agency detection hypothesis being well supported. Those who do not default to agency-based explanations of natural phenomena are at the tail of a number of psychological curves, but they aren’t either pathological or deviant in a real sense.

    Agnosticism is much harder than atheism to maintain. It is perhaps the least stable ideological stance to take (albeit the most honest!)…

  • https://plus.google.com/109962494182694679780/posts Razib Khan

    #6, first, don’t talk to me like i’m an idiot. for example:

    Finally, I’ll point you to one part of cognitive psychological research where recent research showed that effects are not robust at all: It’s the association of intelligence and SNPs, where it turns out that a large scale study could not replicate any of the previously reported results

    i get this quoted back at me about once a week for some reason. should i just wear a t-shirt that states that i am a friendly electronic acquaintance with the first author, and am therefore quite aware of this research? :-) he claims he reads this blog! can i please not have this research cited at me constantly as if i’m going to be shocked anymore?

    back to me not being an idiot: Second, the approach to take a random paper on an esoteric/non-mainstream topic from a not really flag-ship journal to start the discussion about the validity of cognitive psychology is very questionable, to say the least.

    i keep track of the issue related to publication bias in psychology


    and statistically oriented research more generally


    if you read this blog you would be aware that i’m pretty much alluding to these problems in the post. if you didn’t read the blog then, that’s fine, but please don’t assume i’m an ignorant idiot as a null hypothesis. it wastes my time, and it wastes your time.

    thanks for the other pointers!

    p.s. another weird thing is that people ‘explain’ WEIRD to me as if i’m going to be totally enlightened by it. but i’ve actually been a fan of joe heinrich’s work back to the mid-2000s, so i’m quite aware of this too:


  • https://plus.google.com/109962494182694679780/posts Razib Khan

    #7, we’re getting into semantics. i didn’t say pathological. but i did say deviance. because as you say, these people are at the tail of the distribution. if you don’t like the word, feel free to substitute.

    also, i think agnosticism and atheism address two different questions. i don’t know if they’re comparable. as for difficulty to maintain, the technical definition of agnosticism is difficult. but the way most self-styled agnostics perceive the term (i.e., uncertainty) isn’t too hard i think. it’s more like splitting-the-middle.

  • Justin Giancola

    Let’s say you were presented with a proposition on a subect that was totally foreign to you, and you said ‘I don’t have enough info/competancy in my thoughts on this: that lack of knowledge/comprehensibility (even interest) is yours, not what we, the consensus of humans and our endevors, are provided with… So, that limit on what can be known is yours for the moment, and relative to your belief – either intuited or prefered. For you to say, ‘I am not sure if I could ever come to reasoned or inspired conclusion’ is equivalent to the “limits of knowledge” (in a Descartian/skeptic fashion), as you could acknowledge that someone may be able to know this, but I can’t, and without any grasp on the subject, would be an appeal to authority.

    I’d say, the other interpretations naturally flow naturally from Huxley’s own definition.

  • Tapani Riekki

    Few comments:

    “From talking to many other atheists I have come to the conclusion that Atheism is a mental deviance.”

    Point here is that what is deviance? I would say that the far ends (highly atheist, highly paranormal believers / religious people) are deviant but because I’m not sure how pathologically you use term mental, I would just stick with the deviance/variance (individual variation in “cognitive make up” which affects which cultural beliefs / non-beliefs we are prone to adopt).

    “But to me the above theory should make powerful predictions in terms of effect size.”

    The problem is that the phenomena, for example belief, is not explained by one but many individual factors and has a lot to do with social and cultural properties. For example if you consider the decline of religiousness in western Europe happening at the moment or if you consider the decline and raise of religiousness in ex-communist countries in eastern Europe. These fast changes show that individual differences, which probably have some genetic components, that may affect beliefs are highly interactive with the cultural atmosphere.

    Thus, because there is many individual factors that contributes to tendency to belief or not to belief and there’s probably high interaction with the social/cultural properties, you will not get big effect size with one simple psychological test. I would be extremely surprised if a perceptional task could predict extremely well who is believer and not. A bit same thing if you would believe that single gene could predict IQ robustly at individual level. This also means that predictions in the individual level should rarely been done based on group differences – in case of beliefs it is easier just to ask :). However, the point here is not to predict who is religious / paranormal believer and who is not – but rather understand the cognitive mechanisms that play a role in how such a complicated phenomena as beliefs are shaped.

    What comes to powerful predictions, I agree that “belief is a naturally evoked consequence of the general architecture of our minds.” is too simplistic as there are so many non-believers who still have generally (although there’s plenty of results that not exactly the same) the same architecture as everyone else. Better wording would be something like “general architecture of our minds provides fertile ground for / does not prevent religious/paranormal beliefs. Or the other way round: lead automatically to atheism/skepticism.

    It should be also noted that believing in paranormal and religious beliefs are not the same although they share similarities (at the beliefs them are rather similar, for example belief in a christian god or a godly spirit of the earth). Religiousness is much more social phenomena and often measured differently than belief in paranormal (going to church vs. believing in ghosts). Thus, individual psychological properties probably affect less religious beliefs and affect more paranormal beliefs.

    “which results are robust and real, to the point where you would be willing to make a serious monetary bet on it being the orthodoxy in 10 years? ”

    Again, what is robust and real. If we discuss on individual level (because there’s a group differences in the group level, this usually does not lead to good accuracy at the individual level), not so many. If we discuss at the level of phenomena, many. For example I could bet that strong believers in paranormal have a lower criterion value in general, as research using various stimuli has shown. Another example to highlight the point of individual / phenomena level: I would bet that schizophrenics have cognitive deficits (for example with executive functions) also in ten years – although these cognitive deficits are not so specific that you could make robust predictions at the individual level.

    best regards,

    Tapani, author of the paper discussed and reader of the blog

  • https://plus.google.com/109962494182694679780/posts Razib Khan

    #11, thanks. i would say that the strongest point you make, which i’ve wondered about before, is inter-cultural variation. in a place like china atheism is almost normative. not so in arkansas. the prediction i make is that this sort of test will show much bigger gaps in arkansas than in china (in part when atheists are 30-40% of the population they’re going to be more representative).

  • Tapani Riekki

    #12, Yes, the intercultural variations is important point as even though the culture may be hostile to or pro religious, there are always still people who believe / do not believe. You would guess that the ones opposing the norms could have something in the individual level that makes them go against the norm, which of course can also mean that they form smaller groups inside the main culture that agree with their views.

    The questions whether the gaps in Arkansas would be bigger than in China is an interesting one. My prediction would be that the people in the middle will mostly go towards the cultural norm while the extreme ends will do this less. In other words, in highly atheistic cultures the individual differences have less effect on who is an atheist and who is not and religious cultures the opposite. This would mean that in China, the believers have stronger individual features that are generally associated with proneness to believe and in Arkansas, the atheist have stronger atheistic individual features that have been connected to skepticism. Also, this would mean that the floor/ceiling effect hides the individual differences in the group that agrees more with the cultural norm than does not. Hence, it would be harder to separate strong believer from a “norm” believer in Arkansas and a militant atheist from agnostic in China) just by asking do you believe or not or by using questionnaires to do this.

  • Tapani Riekki

    #6, one small comment: even though you may not like the topic of the paper, as it is applied cognitive psychology and not general / or specific to cognitive process psychology, please check the reference list before jumping to conclusions: the research is based on similar research (though sadly most of it has not been empirical) that has been published in for example TICS and psychological review… Anyway, I understand your burst in face of belittlement but still had to point this detail.

    And more generally, what comes to robustness: replicate replicate replicate. Though it is hard to replicate if you do not have something to replicate.

  • CouriousScientist

    I sure didn’t intend to talk to you like an idiot. (Your seem are generally reasonable to me, there was one clear exception to which i posted a scathing response that was never published (because of technical problems or because you didn’t like my aggressiveness – I have no idea … ).

    But I’m also sure we are wasting our time when engaging in discussion, because you already seem to know most of what I’m writing (even if you don’t include in your posts. to me a link to the Nature piece about kahneman’s letter would have been a much clearer entry, even if it is more about social psychology than cognitive psychology).
    Moreover, had I seen Steve Sailer on your blogroll earlier, I wouldn’t have engaged you in the first placed. I don’t tink that I can say anything of interest to someone who reads Steve Sailer on a regular basis.*

    Take care

    *I’ll just quote SS from a blog entry to which your site links right now.
    “My bet would be that Obama’s SAT score was rather like his GPA: fine, but not spectacular. He didn’t make either National Merit Scholar or the black version, National Achievement Scholar, so we can assume he didn’t ace the PSAT. Maybe he was less stoned the day of the SAT? Say, 690 V, 550 M (pre-1995 recentering).”
    “Maybe he was less stoned the day of the SAT?” Good point, I hadn’t though of that! Seriously though, even when I quoted a particularly stupid section, SS’s obsession with race and intelligence is just beyond me, and I would have hoped beyond mainstream publications.

  • http://www.gwern.net/ gwern

    I didn’t notice the effect size point, but I’m not necessarily sure it matters.

    Why? Because the general model (a propensity to over-active agent detection which snowballs through confirmation bias and motivated cognition) doesn’t predict large effects of very firm belief: if it did, we wouldn’t see tons of deconversions or reconversions on a national scale (see the Pew survey), nor would we see that very little kids always start out believing in gods and supernatural beings (despite sometimes growing up to be atheist adults; lots of papers in this vein), nor would we see when an anthropologist like Luhrmann (2012) investigates charismatic evangelicals that only a minority claim outright Godly experiences and then usually only a few times in their entire life.

    What we see is that atheism-theism is a line which is easy to wander back and forth across – which is consistent with a multitude of weak yet statistically significant results.

  • http://www.tdaxp.com DrTdaxp

    Contact me if you want to make a $1000 bet on these issues.

    1. OCEAN big-five personality variables will appear in a greater percentage of the top 100 impactful cognitive psychology academic articles than Meyer-Briggs.

    2. The self-efficacy construct will appear in a greater percentage of the top 100 most impactful cognitive psychology academic articles than “multiple intelligences.”

    3. The “working memory” construct will appear in a greater percentage of the top 100 most impactful cognitive psychology academic articles than “multiple intelligences.”

    Basically, I’m betting with the orthodoxy against two popular pseudoscientific theories.


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Gene Expression

This blog is about evolution, genetics, genomics and their interstices. Please beware that comments are aggressively moderated. Uncivil or churlish comments will likely get you banned immediately, so make any contribution count!

About Razib Khan

I have degrees in biology and biochemistry, a passion for genetics, history, and philosophy, and shrimp is my favorite food. In relation to nationality I'm a American Northwesterner, in politics I'm a reactionary, and as for religion I have none (I'm an atheist). If you want to know more, see the links at http://www.razib.com


See More


RSS Razib’s Pinboard

Edifying books

Collapse bottom bar