I know. I was suggesting you might like to experiment with a different method, for once.

Statisticians tell the tale of The Emperor of China’s nose.
The Emperor of China lived in the Hidden City, where nobody had ever set eyes on him. A statistician was curious about the length of his nose, and could not go to look, so he conducted a survey of all the people in China to ask them how long they thought his nose was. None of them had seen him either, but each made their best guess, and the statistician collected the masses of data – literally tons of evidence – and averaged them. He reasoned that while each individual estimate was probably quite inaccurate, subject to experimental flaws and errors, if he averaged them to find the consensus opinion, the errors would average out and he would get an accurate number – given the number of people in China giving him data, very accurate.

You and Jody say that the psychological research gave reason to expect this, but is that research any better? Suppose they had the same sampling problem? We ask for the evidence, and are pointed to one paper, we say the evidence isn’t there, we are told it is in earlier papers, we chase the earlier papers and find inconclusive results they say need further work to confirm, and asking for the confirmation we are directed to the first paper again. We end up playing the cups-and-ball trick: whichever cup you turn over, the ball – the evidence we seek – is always under another cup.

Does the sheer number of cups, each with the possibility of a ball under it, count for anything?

My interpretation of science does work in the real world; it is possible, I admit, that it isn’t working in academia. You just try getting sloppy work like this passed in pharmaceutical development, or safety engineering, or something with a $100m investment riding on it! In the real world, with lives and fortunes at stake, you do it right or you’re out the door.

Whether or not you can eliminate all variables and flaws, you should eliminate all the flaws that you can. If I as a non-specialist can spot these problems in a matter of minutes, those in the field certainly should have. If I can suggest improvements that should have been made, then what excuse can there be?

The points I raised do question the ‘goodness’ of this science. If you can’t eliminate alternatives sufficiently to come to a dependable conclusion, then don’t waste our money doing the experiment. I can get a “maybe” far more cheaply than that.

#23,

You ought to be able to subject it to your own withering critique, now that I’ve given you an example.

- What’s the sample size?
- What indications are there that the sample is representative of the general population?
For example – they mention the sex ratio, and you can count up how many liberals versus how many conservatives they had from the figure 1a near the bottom. (presumably 0 on the scale is politically neutral.) Are these the same as in the general population do you think, and if not, what does that imply about the sampling process?
- What’s the overlap between the distributions for liberal and conservative? (fig 1a again.) Is the difference in the bulk of the points, or the outliers? How many outliers are we talking about? If there are more points sampled on the left, would you expect the outliers to have the same spread on the left as on the right?
- What other variables and possible alternative explanations did they control for?

Have a go! Let me know what you come up with.

(Incidentally – I have no objection to the idea that liberals and conservatives might think differently for innate reasons. I would expect – like most nature-vs-nurture issues – that it is a bit of both. But I’m highly dubious that the nature of the difference is anything so crude that it will show up in the gross anatomy.)

http://lcap.psych.ucla.edu/pdfs/amodio_natureneuroscience07.pdf

You are also very wrong about hypothesis formation. An experiment in no way requires you to come up with, or eliminate, all other hypothesizes. OTHER experiments do that. The scientists should try to eliminate as many variables as their always limited resources and time allow, and explain any known flaws honestly. But no experiment will ever be complete or perfect.

All of this has now gone into our radically different perspectives of science: pragmatic vs unattainably idealized. But nothing in your statements has done anything to question the science of the specific experiment. It is good science. Like all science, it is incomplete and imperfect, should be retested, and should be examined from other possible hypothesizes. But it is still good science, and worthy of note, citation, and notice.

You’re OK on setting a question and gathering evidence. But when you form an explanatory hypothesis, what you should be doing is forming all possible explanatory hypotheses, and then doing experiments to eliminate them, ideally leaving only one.

You also need to consider all the other consequences of those hypotheses to see what else they predict – partly because the cases where they make different predictions are potential experiments, partly because you may find predictions contradict previous findings. (For example, the hypothesis that brain volume is directly related to cognitive ability runs straight into the facts that men’s brains are bigger than women’s, and sperm whale brains are bigger than men’s. Size isn’t everything.)

You then need to design your experiment to test the alternatives, eliminating all but one. The design has to take account of errors and uncertainty, sources of bias, statistical power, measurability, the statistical analysis to be done (very important to define that before seeing the results), and so on. It’s not a straightforward matter.

Then you perform the experiment – documenting everything, including unexpected problems, according to the design. Perform the tests on the results that you defined earlier. Then go through the design, in light of the results and problems observed, and determine whether the design met the requirements (excluding bias, sufficient statistical power, etc.). And then round the loop again. When you’re sure you’ve eliminated every possible problem you can, write it up and publish. Then wait for other scientists to point out things you might have missed, and then go back and do it again. It is a difficult and painstaking process.

“You constantly write as if any experiment that is not perfect is therefore flawed and should not be considered.”

You can consider it, but you should consider it knowing what its flaws are. Until the flaws have all been removed the work is not complete but still in progress.

“No experiment is ever perfect or conclusive, it is the cumulative weight of multiple imperfect experiments,…”

Be very careful here. The cumulative weight of evidence from imperfect experiments can succeed if the flaws/errors are statistically independent. Systematic errors do not cancel out through repetition, and correlated errors cancel more slowly than you expect. Biases (cognitive and other) tend to be systematic. It is very dangerous to combine partial evidence without an understanding of how the errors are related – another excellent reason for knowing what the potential flaws in an experiment are.

“…and the opinions of scientists in the related fields”

No, definitely not. What matters is whether they have any arguments that can show your results to be potentially flawed (and whether they have checked thoroughly enough to tell). Opinions are irrelevant, except to the extent that they are based on understanding of the actual evidence.

The practice of science is never perfect, but that’s no excuse for ignoring the imperfections and pretending it is. Science progresses through the ruthless elimination of flawed hypotheses to leave only the fittest survivors, like Natural Selection. Science progresses through criticism.

Um, no. The whole point of this was that nobody had taken the conclusions of previous psychological studies and examined them from the perspective of brain mass. The previous evidence was conclusive enough to create an experiment testing for causes.

A review of the scientific method, pulled from Wikipedia. 

1. Define a question.  Is there a relationship between the brain’s formation and the psychology of those self-claiming a political ideology. 
2. Gather information and resources (observe). They examined the studies on the psychology of lib/cons. 
3. Form an explanatory hypothesis.  Mass differences in the amygdala and ACC might account for the psychological differences observed in the studies. 
4. Perform an experiment and collect data, testing the hypothesis. The MRIs, drawing from the pool of available subjects.
5. Analyze the data. Done.
6. Interpret the data and draw conclusions that serve as a starting point for new hypothesis. The results seem to confirm, and do not dispute, the hypothesis.
7. Publish results. Done.
8. Retest (frequently done by other scientists). Here’s where you would come in, if you were a scientist. 

Nobody at this point is pre-filling out voter registration forms based on MRI scans. If subsequent experiments disprove a relationship between amygdala/ACC mass and self-identified political affiliation, it doesn’t mean the science was bad or the scientists inept, it just meant the hypothesis was wrong. You constantly write as if any experiment that is not perfect is therefore flawed and should not be considered. No experiment is ever perfect or conclusive, it is the cumulative weight of multiple imperfect experiments, and the opinions of scientists in the related fields, that progresses scientific theory.  That is why peer review and publication are important.  

While it is true that it is difficult to get large sample sizes for an MRI study, whether it is good or not depends purely on the statistics. An opinion poll based on 90 people wouldn’t fly even in the general media, and one that draws important conclusions from a 14 participant sample less so.

Priors can be a consideration, but we would need to examine the prior evidence for them first before we can make a judgement. If the evidence had been conclusive, we wouldn’t need the MRI scans, yes? We need to be careful of confirmation bias, and not judge the present study more leniently because it fits in with our preconceptions.

There are a whole lot of other assumptions they’re making – for example, that the political test only tests politics and not anything else, or that brain volume is positively correlated with strength of cognitive or personality traits. It’s depressingly reminiscent of 19th century IQ tests and craniometry, which was also used to justify beliefs about out-groups. Even with simple systems, you have to be incredibly careful to be able to be really sure that you’re measuring the right thing, and not fooling yourself that you’re seeing what you expect to see, and when it comes to something as complicated as the human brain, the alternative possibilities explode. They’re not being nearly careful enough to be able to derive a result of this nature.

It’s definitely not postmodern, but it might be Cargo Cult science. (You’ll recall that Feynman specifically used the example of psychological research for illustration in his essay.)

Yes, they discuss sampling procedures (in enough detail to raise concerns), but there’s a lot we don’t know about it, or its possible influences on the study. Why, for example, that odd lack of conservatives? It’s irrelevant that it was peer-reviewed and published – peer-review frequently doesn’t check details like this, all it says is that this study is worth reading, not that it is any good.

Chris may well have been being flippant about Colin Firth – I would imagine that the study was done as part of a BBC Radio 4 programme in which he participated – but it’s a long-standing point between us that science is routinely being judged on the basis of Argument from Authority: the qualifications and expertise of authors, the prestige of peer-review journals. I was being somewhat flippant myself in pointing out that once again he was basing the credibility of a paper on its list of authors.

Yes, there are many possible alternative scenarios to explain the results, but as clearly stated, the experiment was building on previous psychological studies about the differences between liberals and conservatives. It was designed with a hypothesis based on those previous studies: that brain mass in certain key areas could account for certain, already observed, lib/con behavior. The areas of interest have previously established effects on human reactions. Your analysis presents it as if they just measured some brain mass and pulled a conclusion out of the air.

Their study was based on previous work in the field, and the results seem to confirm their logical hypothesis (and don’t seem to contradict it). They clearly and openly discuss the selection criteria and other potential mitigating factors. It was then peer reviewed and published. Even the authors don’t claim it’s conclusive, and you can feel free to test your own hypothesizes. I believe Chris’s point was that this is perfectly sound science, and nothing you said challenges that even remotely.

I don’t want to put words in his mouth, but I believe Chris was being flippant about Colin Firth because it was funny, and this is a blog, and as usual you should lighten up. Also, because the person he was responding to blithely called it postmodern junk science (which even a quick scan debunks) and so it didn’t require more than flippant dismissal.