“I Just Don’t Believe Those Results”

By Neuroskeptic | August 7, 2016 5:28 am

Are some scientific results so unexpected that we should just reject them?

This is something I’ve been wondering recently. It’s one thing to disbelieve a study because there are problems with the methods used. But is it scientifically valid to judge a study by its results alone, even if you don’t know of any methodological flaws?


I’ll admit it – I do judge studies by their results. The most recent example of this was this study which I read about yesterday. Briefly, the study reports that priming men to think about gender (by asking them one question about their spouse’s wages) causes a 24-point swing in favor of Donald Trump over Hilary Clinton.

I just can’t believe this result. I don’t have any criticisms of the methods of the study, but I simply don’t believe that such a subtle prime would have such an enormous effect. For comparison, since the Democratic National Convention, there has been roughly an 8-point national swing in favor of Clinton; this is widely regarded as a dramatic change. Yet according to the priming study, one single question can provoke a swing three times as large as that (in men). I don’t find that plausible.

Yet is my skepticism about this result justifiable? Isn’t there a sense in which it’s unscientific? If I’m free to decide that a result is just unbelievable, how am I any different from (say) a creationist who maintains that it’s just too incredible that natural selection produced humans and other life? To put it another way, how can I call myself a scientist if I sometimes reject scientific evidence that conflicts with my intuitions?

On the other hand, maybe my skepticism is perfectly reasonable. It’s a fact of life that experiments sometimes go wrong, and give the wrong results. Often, it’s the unexpected or impossible nature of the results that first alerts us to the problem. If I measure (say) my own weight and the scale says I weigh 1 kg, I would immediately know that my scale is broken or miscalibrated. Some results really are just wrong.

In Bayesian terms, we would say that my prior probability of a 24-point priming swing is very low. If my prior is low, it is perfectly rational for me to remain unconvinced after seeing one study in favor of the enormous priming effect (it might take ten such studies to convince me). My concern, however, is whether I can justify having such a low prior?

If I had to explain my a priori skepticism I’d have to fall back on vague, intuitive statements such as “small influences have small effects” and “you just never see swings that big” – which sounds an awful lot like hand-waving, and brings us back to the problem of creationism, amongst other things.

Is there a solution? Is it possible to be purely scientific in our evaluation of scientific evidence, or will there always be an element of intuition?

CATEGORIZED UNDER: science, select, statistics, Top Posts
  • http://jayarava.blogspot.com Jayarava

    Skepticism is *always* justified in the first instance, especially for results with low prior probability. I always go back to Hume on this one. For reports of a miracle to be credible, it would have to be even more miraculous for them to be false. Or to put it another way, strongly counter-intuitive results require stronger evidence.

    If someone comes up with the result that says they produced fusion at room temp or that neutrinos were going faster than the speed of light, then yes, we are all skeptical.

    In this case priming has been in the news a lot lately, as you’ll know. And the very existence of the effect has been called into question. So claiming a massive priming effect right now is more than a little bit suspect.

    You link to a journalistic interpretation of the study. But I don’t see a link to the study itself. So skepticism is more than justified.

  • kaleidocyte

    Creationists often dismiss evolution by appealing to their personal incredulity or ignorance, which is a fallacy. Presumably you wouldn’t do that? Instead, you would question the finding on other grounds, or you would suspend judgment and seek stronger evidence regarding the effect. Importantly, you’d presumably be open to revising your position if, as it turns out, the effect is shown to be durable rather than anomalous. The same is not true for many creationists. Consider Ken Ham as an example. In his debate with Bill Nye, when asked what (if anything) would ever change his mind, Ham basically said that nothing would; he regards his religious intuitions as infallible.

    • disqus_atlq8Zmtsd

      I am not familiar with that debate, but I have to point out that many find “religious experience” to be a very real piece of qualitative evidence. I think many could and probably have argued that their personal religious experience is a stronger piece of evidence than any piece of “scientific” evidence could be.

      And I certainly understand that influence. Of course, I think it doesn’t have to be either/or. There is zero reason that belief in intelligent design and acceptance of the big bang theory/evolution can’t coexist.

      • kaleidocyte

        Yes, some religious folk seem to regard their personal religious experience as infallible, and therefore “stronger” than anything that would bring it into doubt. However, this doesn’t help to establish the truth of their claims.

        • disqus_atlq8Zmtsd

          I only said it was justifiable, not persuasive.

          • kaleidocyte

            In what sense though? If Ham appeals to his personal religious experience as justification for claim X, and Ham’s debate opponent does the same for claim Y, which is incompatible with X, then it would seem we still have a long way to go toward figuring out who is right, and why.

          • disqus_atlq8Zmtsd

            Wouldnt that play into it not being persuasive?

            My personal religious experience can be a valid reason for me to claim X. Your personal experience can be a valid reason for you to claim Y. The inability to share those “results” make them poor fuel for a debate.

            So yeah, in a formal debate setting religious exoeriemce serves as a fairly worthless appeal. In an informal discussion though, it cannot be disregarded if the goal is to persuade the person making the appeal in the first place.

          • kaleidocyte

            We cannot disregard it, but how do you go about persuading a person who insists, from the outset, that nothing you present would ever be enough to convince them otherwise? How do persuade someone whose religious intuitions are so entrenched that they dismiss anything that would bring them into doubt?

          • disqus_atlq8Zmtsd

            Many theological approaches encourage challenging your beliefs in order to hopefully (by this way of thinking) strengthen them.

            If someone is unwilling to do this, we must assume them unreasonable, uninterested, or ultimately enlightened. :)

        • http://oldephartte.blogspot.ca/ opit

          When sales presenters are trained, they are carefully coached in a scenario which they are to replicate. Their uncritical acceptance is important as they communicate it to their contacts. But : there is no attempt at impartial truth. When discussing religion it is important to remember that it is a mass appeal of definite thought limiting agenda by sales people of varying competence.

      • MCope

        Religious experience is only evidence of religious experience.

    • djlewis

      Was Nye asked that question? 😉

      • kaleidocyte

        Yes, and his answer was evidence.

        • djlewis

          Of course. But what kind of evidence — how much — etc? My point is that scientific “theories” are *all* actually provisional until further evidence appears that falsifies them and maybe leads to a new theory (but maybe not). That is the nature of science.

          What distinguishes so-called “true” theories from speculative ones is the weight of evidence for the “true” one. For evolution that weight is enormous, presenting a very high barrier to falsification, but not an absolute one.

          Hey, there was a helluva lot of evidence for Newtonian mechanics until it was falsified by relativistic mechanics, first with the Michelson-Morely measurements, then as a new theory by Einstein and others (yeah, others), and then with lot of new observations to validate it.

          So what distinguishes pseudo-science theories from “real” science theories is not two different meanings of “theory” but the possibility of falsification in the zeitgeist in which the theory is embedded. But actually, today the zeitgeist surrounding science itself is sceinTISM, which is not only NOT science, it is anti-scientific, that is, admits no falsification of its own!

          • kaleidocyte

            If I recall correctly, Nye offered some examples of the kind of evidence that would bring evolution into doubt. The point was that, for Ham, the evidence is dismissed out of hand if it contravenes his religious beliefs.

          • OWilson

            Evolution is a property of life, not a cause.

            A pebble doesn’t grow into a mountain.

            To me, both bible thumpers and Big Bangers with their absolute certitude, are the same kind of “believers”.

          • kaleidocyte

            I don’t see how, given that (1) the Big Bang wasn’t the topic under consideration in the Nye-Ham debate, and (2) the Big Bang is well supported, whereas the core claims of young earth creationism are either unfounded or demonstrably wrong.

          • OWilson

            The Bible was “well supported” for a long time, too!

            So tell us, what caused YOUR particular Big Bang? :)

          • kaleidocyte

            If you’re asking how the universe came to be, then I don’t know. And from what I can tell, no one knows. The Big Bang theory, in its present form, explains the expansion and evolution of the universe from an extremely hot and dense state to its current form. But we don’t yet know what processes result in the formation of universes.

          • OWilson

            Thank you!

          • kaleidocyte

            For what?

          • OWilson

            Your comments explaining your thought process?

          • kaleidocyte

            No problem. :)

          • http://oldephartte.blogspot.ca/ opit

            Given that the Big Bang would have destroyed the evidence it means that the theory is, in its own way, a dead end.

          • kaleidocyte

            Also, I’m not sure that you can call it a “working hypothesis” if it doesn’t actually do any work.

  • Sanity

    I agree with your skepticism of this study. But then I am a skeptical person by nature. I was raised up north by progressive parents, and attended progressive schools. However, I have been living in the Deep South for 20 years. So I can honestly say I have met many people who would give credibility to this study So the question is, where was this study taken? If it was taken up north then yes its validity is questionable. If it was taken in the Bible Belt or out west, then it would correctly reflect their beliefs.

  • OWilson

    To a large extent you are dealing with simple confirmation bias.

    The “Experts” all laughingly agreed that Trump was a joke, and couldn’t ever get his 1237 delegate votes. (He finished up with close to 1600).

    Studies tend to confirm what one is actually looking for. This is even more of a problem when a grant is sought from an interested party.

    Political polls reflect their sponsors biases, whether Media or Academic.

    Then there’s the “interpretation” , or “what it means”. which is always amusing.

    The ones behind in the polls, say that folks don’t pay attention until after conventions, Labor Day, or The Debates, you name it.

    We even have posters here who laugh and denigrate religious beliefs, I would point out the a “Grand Designer” makes far more scientific sense that the “belief” that the Universe and we, all came from a big bang from an infinitesimally small that appeared from nowhere. That’s a fantastical miracle!

    (I’m an atheist, by the way) :)

  • Nitric-X

    The author provides the cross-tabulations here:
    As one can see, the post refers to a sample of n=176 vs. n=168 males with or without prime, respectively – a relatively small number – always prone to statistical error. The numbers in the 2×2 read 42,49,50 and 30 (for simplicities sake, and for the lack of any a priori hypothesis I ignored the few “don’t know” and “refuse to take part” answers). If you compute the OR, you end up with a p=0.06. That’s not bad, but not significant either. Taking into a account the low a priori probablilty of such a large effect size (I was too lazy to calculate it…, but assume it would be quite higher than 2.0), the results are likely to be a false positve due to an underpowered study.
    Having said that: I guess what made you skeptikal about the results is lifelong training in the interpretation of scientific studies, so that your “fast cognitive system” indicates that something might be wrong with the study. Just as a chess player intuitively categorized patterns on the board. As you are a scientist, this is the starting point to check the integrity of the study, and if robust, to replicate (or rather falsify) it (in the present case may prediction would be that the priming effect might be there, but it is quite small, and it requires samples magnitudes larger than the one used here. Just a guess.) Creationists don’t bother and stick with “I don’t believe it”…

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      Ah, thanks! So now I have a good solid reason to be skeptical. But I was skeptical before I had a good reason.

      • Nitric-X

        I guess you had a good reason (gut feeling) as a starting point – but this can never be the finishing line. IMHO that makes the difference…

      • http://oldephartte.blogspot.ca/ opit

        It sounds as if there is a Cause and Effect logical difficulty instead. i.e. We believe man is a rational animal rather than recognizing he is a rationalizing one. How is this important ? Because of the need to recognize and reject Logical Fallacies. Subjective assessment of what is reasonable can slide without our recognizing it – and can be attacked without detection on that account. That is the province of Moving the Goalposts aka Moving the Overton Window

  • Steve Politzer-Ahles

    I think it’s fair to be skeptical of a study wholly based on an implausible effect size, like you’ve done here. Gelman & Carlin give a more formal discussion of this idea: http://www.stat.columbia.edu/~gelman/research/published/retropower20.pdf

  • johnLK

    No reason to be timid. I’ll quote Carl Sagan saying, roughly, what others have said, “extraorinary results reqire extraordinary evidence.”

  • https://www.facebook.com/pages/My-Original-Music-written-arranged-produced-by-ME/195887277117017 JohnnyMorales

    Maybe you’re skeptical, because you evidently only read the headline, and didn’t even bother to read the summary explanation.

    If you did, you’d find out the same set up was used in regard to a Sanders/Trump match up.

    It produced very different results.

    Sanders numbers were barely affected by the question.

    For whatever reason you minimize the impact the question they asked by NOT even stating what it was.

    It was a LOADED question designed to hit home with the men who MIGHT be sensitive about the answer.

    Not all men would be sensitive, but enough were to produce the results.

    Of course the results are probably overstated, but not to the degree that the opposite is actually true, or that the results have no basis in reality.

    Sexism is rampant in this nation, and the notion that a significant percentage of men automatically have negative feelings about women in power shouldn’t be a surprise.

    The fact that you reacted so strongly to general evidence in support of this is certainly odd.

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      I’m aware of the Sanders/Trump part of the experiment but it doesn’t bear on my point that the effect in the Clinton/Trump matchup is enormous and far too large to be believable (for me).

      I’m not saying the true effect size is zero, there may really be a smaller effect (as you say, “the results are probably overstated”), all I’m saying is I don’t believe it is 24 points.

  • James Downard

    Intuition & background can’t be teased out of the mix that keeps the social dynamic of science running along. When the hoaxed Archaeoraptor fossil was promoted by National Geographic back in the 1990s, it was ornithologist Storrs Olson at the Smithsonian (one of the core group digging in their heels on the dinosaur-bird paternity trail) who smelled a rat & helped expose it as a chimera. Ironically, half of the concocted fossil turned out to be from Microraptor, a four-winged critter that was much more interesting than the chimeric faked fossil, helping to clarify early dino fliers in the Cretaceous. The science moves on, and that takes place with the mix of human foibles & motivations as we find them.

  • Pingback: Sunday Assorted Links | Marginal Counterrevolution()

  • Nofun

    This must be like where conservatives refuse to believe low unemployment figures.

  • Erik Bosma

    The biggest difference between your skepticism and, say a creationist’s, is that you put your doubts up on a public forum where any and all could give you input. A creationist would NEVER do that. You need to add open-mindedness and closed-mindedness to the mix.

  • Rebecca Weinberg

    It isn’t a 24 point swing. As described in the article you linked, it’s a 24 point swing among likely male voters in NJ, which is partially compensated for by likely female voters in NJ (i.e. it could be costing Clinton 8 points overall). 8 points is well within the range of what we see in different polls, so I don’t see any a priori reason to think this is unreasonable. I’m glad to see the sample sizes Nitric-X mentioned, so that we can put it into a real context. Given that and the 8 point interpretation of apples-to-apples, it seems very plausible this specific poll result might have been on the high side if we reran the experiment. That said, my best guess is the basic effect is real.

    Although the priming effect isn’t one I have trouble believing, there are claims I hold to a higher degree of skepticism by default. Usually when they seem too delightful- like “chocolate fights cancer” type claims. The most recent headline I encountered like this was “breastfeeding is associated with baby’s longer telomeres”. I *want* to believe this, but between my existing skepticism of breastfeeding literature and my existing skepticism of telomere literature, I would be surprised if this finding is reproducible and robust enough to have a biological impact.
    Some of my skepticism is a self-protection thing, some should probably be general (i.e. some areas of the literature are much more likely to be reported on with more limited data available and that should probably go into Bayesian priors). But for me to say definitively, “I just don’t believe those results”, it’d have to be pretty extreme. Teleportation of humans might do it. Non-microbial alien life perhaps.

  • KleRoi

    Being skeptical is exactly what makes you a scientist. The difference between you and a creationist is that you don’t believe something that has very limited evidence (1 study), while the creationist doesn’t believe in something (evolution) that has a mountain of evidence to back it up, or alternatively he believes in something (creationisms) that has zero scientific evidence behind it.
    A true scientist is skeptical, very skeptical. I might be an extreme case, but I don’t believe in anything I read or see anymore unless it has a LOT of data behind it, multiple replications, etc.

  • Gene Starwind

    Exit polls USED to be excellent predictors of the results of elections, until electronic voting became the norm. Then the exit polls became quite useless.
    Simple: electronic voting machines are rigged. Occam’s Razor.

  • Hosni

    The study measured an enormous % impact on male voters. The 24% number cited here is huge, but other less notable impact measurements from the study are large, too. So additional studies are needed to support these findings, or to qualify them. Is this purely an election-year phenomenon? Due to limiting the sample to men in New Jersey? Would information pertaining to the male’s income or to the condition of his marriage affect the findings?

    About 1/3rd of the swing of male voters from Clinton to Trump also occurred when the survey asked about a Sanders-Trump matchup, which is *not related to gender. That leaves about 2/3rds of the measured impact for “male insecurity.” The 24% impact figure is an inflated number.

    The researcher attributed 100% of the swing in voter preferences in the survey to “male insecurity” within the family as opposed to any other correlated explanation. It is also possible that when men are reminded of their depressed economic circumstances, they favor a protectionist President Trump over a relative free-trader President Clinton.

    … Regarding your faith in science: I cannot recall any scientific principles or predictions that have stood the test of time — without being revised, corrected or replaced. So faith in scientific findings is misplaced. Our faith should only be invested in the process itself. Written history records that the search for truth began with Adam and Eve, who paid the ultimate price for being curious. All humans share this trait.

  • ohwilleke

    Extraordinary claims call for extraordinary proof.

    • Boogie Ondown

      I’ve never seen an “extraordinary” proof, just plain old facts.

  • Lee J Rickard

    This is a big topic in epistemology right now, especially as it relates to understanding when and how you could identify misfeasance and malfeasance in research results. There’s a good recent discussion in Hanne Andersen’s ‘Epistemic Dependence in Contemporary Science,’ in the collection “Science after the practice turn in the philosophy, history, and social studies of science,” edited by Lena Soler and others (2014).

  • davidj

    I think it could be helpful to consider different levels of “accepting” results:
    1. You are convinced/skeptical
    2. You expect most other people to be convinced/skeptical

    It seems more or less fine to judge some studies by results on the first count. The second usually demands a better level of evidence.

    These different levels of conviction get mixed up, though: I’d guess most people want to believe things which are publicly defensible, so it’s often helpful to demand better evidence even on the first level. On the other hand, some people are authorities, so whether or not they believe something has a big influence on whether or not other people believe it.

  • bwana

    I would rather substitute prejudice for intuition. No matter how hard we try there will always be a prejudice towards various results, depending up on the researcher.

  • D Samuel Schwarzkopf

    I think your post hits right at the center of the debate about Bayesian statistical philosophy. No, it is not unscientific to reject claims (and results) that are implausible. And it is not unscientific for two researchers to differ in their prior beliefs about a particular hypothesis. They may come from different perspectives and place different weights on previous evidence.

    What is unscientific is the notion that there is an objective way to evaluate evidence. Even if you do what some people claim to be objective, you are using an implicit uninformed prior – in some ways this is far less defensible because it places undue faith on completely unrealistic observations.

    Whether you call yourself a Bayesian or not, in my opinion we need to finally accept that good hypothesis-driven science requires making clear predictions about the results. Even in the frequentist framework this is needed because you should base power calculations on expected outcomes.

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      Thanks, good comment. But my question is, if there is no objective way to decide on a prior, what’s to stop me having a prior such as e.g. “the chance that all life evolved by natural selection from the primordial soup is 1/10^1000” and then being a creationist?

      • D Samuel Schwarzkopf

        Priors don’t come from out of nowhere but are based on previous evidence. This is precisely why it is actually less defensible to have a uniform prior that assigns equal probability to d=5 as to d=0.1 than one that takes into account that most effects are probably modest.

        Nobody is stopping you from saying that life arising in the primordial soup is infinitesimally unlikely but that doesn’t make it a good prior. The whole point about priors is that if you can’t justify it you can’t blame anyone else for not accepting it.

  • Pingback: Weekend reads: Manuscript submission headaches; Trophy Generation goes to grad school; is science fucked? - Retraction Watch at Retraction Watch()

  • Pingback: Science Snobs Make Us All Stupid – Hayley Is A Ghost Geek | InnerCirclePress.com()

  • Pingback: Science Snobs Make Us All Stupid – Hayley Is A Ghost Geek | We Seek the Truth!()

  • 58554

    Often overlooked is the motivation for stance. If accepting even small amounts of evidence that your position on something would put you out of business, with no back-up, you will impeach the messenger, deny the evidence, spin the data, lie, cheat, even kill if necessary to survive. Presenting evidence to a cleric will not not result in any change to the theology. It might result in a change to a follower of the cleric who is just a little skeptical about some of the a priori premises. I think it is important to emphasize the difference between a premise, a hypothesis and a theory.

  • Franck Ramus

    Physicists do that all the time: they check that their numerical results have a plausible order of magnitude. They are considered really poor scientists if they don’t. You do exactly the same, by evaluating the plausibility of the magnitude of the effect size. I think that psychologists would be well advised to do that more systematically before submitting rubbish results.
    Of course such plausibility-based disbelief should not be definitive, but it should incite scientists to double-check their raw data, their calculations, and to perform independent replications before believing. And reviewers and editors to require such checks.

  • joseph2237

    I would tend to agree since the subject has perplexing meanings to a man’s view of his ego. When you involve ego the out come can mean anything.

  • Jenny H

    I can believe their results — but many times NOT the conclusion.
    Many times they have only shown is such a such condition, we have measured that significant number of men change their immediately stated political view by being primed by the simple question “How much does your wife earn.”
    The conclusion that I would draw from their results is that the opinions of swinging voters tend to swing from the slightest of causes.

  • Pingback: Lectuur op zaterdag: te warm, te goed of te straf | X, Y of Einstein?()



No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.


See More

@Neuro_Skeptic on Twitter


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Collapse bottom bar