Are We All Homeopaths Now?

By Neuroskeptic | February 10, 2010 9:45 am

There are two kinds of Complementary and Alternative Medicine (CAM) – the ones that involve actually doing stuff, and the ones that don’t.

Things like herbal medicine, chiropractic, and acupuncture could plausibly make someone better, as more than just a placebo, given what we know about physics and chemistry, because they involve physically acting on the body. I don’t claim to know whether they do in fact work, but in theory, they could.

Other CAM techniques, however, are just magic. Homeopathy is the best example of this: it cannot work, except as a placebo, unless our understanding of nature is fundamentally wrong. The “active ingredient” in a homeopathic remedy is diluted in water to the point where not a single molecule of it remains (and then diluted more, for good measure). If some mystical “essence” or “energy” can somehow survive in water despite dilution then, logically, all water must contain the essence of pretty much everything. Homeopathy literally involves nothing beyond sugar pills and water

But there’s one useful thing about homeopathy: it shines a light on the rest of modern medical science, or rather, it holds up a mirror to it. Unfortunately, the reflection is not as pretty as you’d hope.

These two graphs come from a paper by Shang et al, Are the clinical effects of homoeopathy placebo effects?, which was a major meta-analysis of 110 randomized, placebo-controlled trials (RCTs) of homeopathy. It was published in The Lancet in 2005.

Shang et al 2005 was bad news for homeopathy, because it concluded that “[the meta-analysis] is compatible with the notion that the clinical effects of homoeopathy are placebo effects.” – i.e., homeopathy doesn’t work. Since its publication the paper has been hotly criticized by homeopaths, and defended by skeptics, with the skeptics generally being right. But it was bad news for conventional medicine too.

These two graphs are funnel plots. Each dot represents a published RCT. Dots to the left of the vertical line are trials where the ‘active treatment’ did better than the placebo control; the further left, the better. The higher up the dot is, the more “precise” the results of the trial, i.e. the less variability there was in the results. They may not look like much, but they’re terrifying.

The top funnel plot shows the 110 published RCTs of homeopathy for various illnesses. The bottom one shows 110 RCTs of “conventional” medical treatments that Shang et al picked out as comparisons. These trials were quasi-randomly selected from a list of over 300,000, such that each one was matched to a homeopathy trial for disease, and number of patients.

You’ll notice that the two plots look rather similar – there’s a lot of spread, but most of the dots are to the left of the vertical line, meaning that the treatments were better than the placebos. Quite a lot are very far to the left, meaning the treatment worked really well. Very few are on the right.

But homeopathic treatments, by definition, are placebos – they’re literally sugar pills. So any trial of homeopathy should have an equal chance of finding it to be better than placebo, or worse. Placebos are placebos. It should be a coin toss, 50/50. In fact, Shang et al found only about 20 trials showing homeopathic placebos to be worse than placebo placebos, and 90 finding they were better.

How can this be? Either homeopathy works, in which case we need to rewrite physics and chemistry, or there is something very wrong with the published literature. I find it easier to believe the latter. But then how could the published literature be so wrong?

Almost certainly, a large part of the answer is publication bias, broadly speaking. If people do a trial and don’t get the result they want, they generally either don’t write it up for publication; or if they try to, it doesn’t get published. Related to this is selective outcome bias: they pick out and write up only those results that do match what they wanted; or they pick out statistical techniques to get the result they wanted, etc.

The plot for homeopathy RCTs is what you get when people study a treatment that doesn’t work, but that they believe does work, and publish their findings in a biased way. But the plot for “real” medicines looks disturbingly like that.

In other words, the whole clinical trial literature – all of those RCTs and meta-analyses, published in respectable journals, the ones we rely on to determine what treatment decisions doctors make – could be produced even if all of our treatments were no better than placebos. Like I said, terrifying.

I should stress that this doesn’t mean that real medicines are no better than placebos. Shang et al’s results are also what you’d see if there’s no publication bias in conventional medicine, and the treatments work really well. (The evidence for positive effects in the “real medicine” trials was also somewhat stronger than in the homeopathy trials – the dots were further left – which is reassuring, but the difference was pretty small.) The problem though is that we can’t tell – at least not on the basis of the clinical trial literature.

Luckily, there’s an answer – mandatory registration of clinical trials. Medical journals or, ideally, governments, can require researchers to publicly announce the details of each trial, and how they plan to analyze the results, before the trial takes place, and require that the final results are made public. The USA has had such a system in place, backed by law, since 2007, and most major medical journals now demand registration.

Sadly, biases still seem to be happening in registered trials. But this doesn’t mean the system doesn’t work, it just means it should be more strictly enforced, and extended to other countries and, I’d argue, beyond just clinical trials. Either that, or we might as well take up homeopathy.

ResearchBlogging.orgShang, A., Huwiler-Müntener, K., Nartey, L., Jüni, P., Dörig, S., Sterne, J., Pewsner, D., & Egger, M. (2005). Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy The Lancet, 366 (9487), 726-732 DOI: 10.1016/S0140-6736(05)67177-2

CATEGORIZED UNDER: papers, science, woo
ADVERTISEMENT
  • http://www.blogger.com/profile/16335234858298947954 ClockworkOrange

    I don't agree with the following sentence:
    “[…] the clinical effects of homoeopathy are placebo effects.” – i.e., homeopathy doesn't work.”
    Studies show that the placebo effect in fact has an effect in improving the individual. So it's wrong to state that homeopathy doesn't work, when it certainly has an effect. The fact that this effect is a placebo effect doesn't diminish that there IS an effect.

  • http://www.blogger.com/profile/08820607553556177222 SustainableFamilies

    yeah I use homeopathy because if a sugar pill can cure me than there's definitely not any side affect, and what better way to cure your body than for your body to heal itself? I consider it an act of giving yourself attention. Healing by demonstrating you know your body needed SOMETHING.

    I was initially worried about it when I heard there were things like mercury in the “cures” but once I was sure that it was diluted to the point that absolutely not one speck of the substance remained (except the energy) I was sure that this was the magic of placebo medicine. But I still think that's great! So long as the treatments are cheap however.

    I know one lady who charging 400 dollars and appointment and that was SO unethical. (Fortunately I skipped the ridiculously expensive homeopath idea and just buy mine at the store or get them for free from my naturopath.) I do believe that in charging that much she probably WAS upping the power of the placebo affect. This was supposed to be a one time, cure all, the cure of all cures kind of thing.

    So in knowing it's 400, it makes one think (if you don't know enough about homeopathy) that this must be the REAL DEAL. It always surprises me when people I know who I thought were well researched use homeopathic ointments and creams and medicines. And they shrug and say, “Oh I didn't know it was just sugar.”

    I like herbal medicine. I think it's kind of dangerous at this moment because some of it could harm the body. (Also if people use it instead of a treatment they really needed.) But I also think there are all sorts of possibilities that will have less side affects than modern science. And I hope that they are scrutinized and deliberated over. But I think there are possibilities there.

    I don't trust western medicine as much as a lot of people. I also remain unsold on “eastern” or “homeopathic” medicine. I think some of those chinese herbs are outright dangerous. (I happen to like acupuncture, but it's probably just because of the pretty music and people whispering in soft chinese nearby.)

    But the power of placebo is wonderful. I think it would be wonderful if we found a way to add that to our medical repertoire ethically. I.E. telling people, “New research has demonstrated that the power of intention toward healing can actually cure you better than some medicines. Every day spend twenty minutes in prayer or meditation on healing and take this sugar pill which has been demonstrated to heal people through them putting their belief in healing into it.”

    I wonder what the trials would be on that treatment. (I bet a meta-analysis of clinical trials would look about the same as the two above!)

  • http://www.blogger.com/profile/08820607553556177222 SustainableFamilies

    And if it wasn't working, move on to the next phase where uncured people were given three sugar pills, and told that in studies three sugar pills were demonstrated to help more.

    Of course how would you create a control group?

  • cazfan

    Isn't it true that most alternative therapies work on hazy medical problems, so that if you case matched you might expect to fine the standard medicine was also a bit on the hazy side?

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    cazfan: That's a good point. Some conventional treatments just obviously work, like antibiotics for bacterial infections, so much so that it would be unethical to compare them to placebo in a trial. CAM trials won't deal with that kind of disease.

    But even if the problem is limited to hazy bits of medicine, it's still pretty bad, because most people don't have clear-cut diseases, but almost everyone (potentially) kind of has a hazy disease (that's what makes them hazy, really) and hazy diseases are what drive most drug sales.

  • Left Coast Bernard

    Neuroscientist,
    Your conclusion about the research data for scientific medicine based upon the Shang study of homeopathy is flawed in so far as it judges the graph with the label Conventional-medicine trials.
    Researchers do a study to find out if a proposed therapy works. It is hard to find a good therapy. Most proposed treatments turn out not to work. A few are successful. Therefore, from a sample of 110 studies drawn from a sample of 300,000, we would expect that they would show a few that work and most that don’t. Homeopaths’ attempts to validate their field, however, usually involve studies of therapies that they believe work and are in use but just haven’t been tested.
    The difference between homeopathy and scientific medicine is not that the two charts should look different, although it is telling, as you note, that there are no homeopathy trial results with odds ratios less than 0.01. The difference resides in what the researchers do with the results of their particular study. A researcher who did the studies in the Conventional medicine graph with odds ratios below 0.01, I see two of them, and a few more close to 0.01, would conclude that the proposed therapy was likely to be effective. A researcher who did the studies in the region between above 0.1 would make some judgment as to whether further research was justified, or whether she would better spend her time trying something else. A researcher who did one of the conventionally medicine trials in the blob around 1 would think that the new treatment was likely ineffective and move on to something else.
    I believe that this behavior of an empirically-based, scientifically-minded researcher is different than that of a homeopathic researcher. That researcher might conclude that his proposed treatment works just as well as it ever did, and recommend that it continue in use as before the study. The homeopathic researcher, I believe, would not take any result as evidence that the therapy is not likely to be effective.
    There are many cases in empirical medicine in which researchers tested existing therapies and recommended abandoning those therapies when they failed to show benefits in suitable trials. Hormone replacement therapy is a recent example. Are there any examples of a homeopathic researcher recommending the abandonment of an existing therapy when a study showed an odds ratio of 1?
    This brings up a question about the Shang study. Were they comparing studies of proposed new homeopathic treatments with proposed new empirical medicine treatments? Perhaps the premise of their study was wrong. They should have compared studies that were attempting to confirm the effectiveness of existing homeopathic treatment with studies attempting to confirm the effectiveness of accepted empirically-based-medicine treatments. Or they should have compared studies seeking new homeopathic remedies with studies seeking new scientifically-based treatments.
    As for your gloomy conclusion about so-called conventional medicine, what would the comparison of existing conventional treatments with proposed new treatments look like?

  • dearieme

    Since we all drink Caesar's piss, shouldn't that stop us being bald and epileptic?

  • http://www.blogger.com/profile/09586588053165570068 mlsci

    Mandatory registration of clinical trials is a great idea. It's like calling the shot in pool.

    Good write up. Thanks for that.

  • http://www.blogger.com/profile/07415199338332642534 jim

    There's an interesting interview on publicatin bias here:

    http://www.abc.net.au/rn/healthreport/stories/2008/2312092.htm

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    ClockworkOrange: Fair enough. That's what I meant, but I often find myself saying “X doesn't work” as a proxy for “X doesn't work except as a placebo”, which is appropriate sometimes although in some cases can be misleading.

    dearieme: I'm neither bald nor epileptic plus, I've never led Roman legions into an invasion of Britain=, so I think it worked.

  • Anonymous

    And why are you so afraid to ADD to known laws of physics and chemistry (not to rewrite them)?

    If one BELIEVES in homeopathy, using homeopathic treatement will release the healing forces inside the body. Through conciosness of the person. You don't need to rewrite laws of physics and chemistry. You just need to see that other laws can rule the physical laws as well and it's not *just* placebo effect, it's not just sugar pill. That pill comes with a believe in homeopathy and a pill without that believe would act differently. Clearly homeopathy can't act through known laws of physics and chemistry but what about other reals?

    When quantum mechanics came, we needed to change how we think about physics. Fortunately we didn't need to rewrite old physics – just to add a huge chunk to it and show that old theories could be derived as a limit case of the new ones.

  • http://www.blogger.com/profile/06832177812057826894 pj

    I wonder what you would expect a funnel plot of RCTs to show if there were not publication bias?

    I'm not convinced that a funnel plot has much meaning anyway, but a funnel plot of any conventional medicine RCTs – well what exactly are we expecting to see? In the same way, if we did a meta-analysis of all conventional medicine RCTs what would that tell us? I'd like to suggest, not very much.

  • http://www.blogger.com/profile/06832177812057826894 pj

    Also – the funnel plot for homeopathy doesn't quite show what you are saying. Strictly speaking all a funnel plot should show for a well behaved (i.e. low bias) dataset is that there is a funnel shaped spread that is roughly symmetrical – the position of the pinnacle of that funnel is not relevant – of course we think that peak should be at a relative effect of 1.0 so that half the studies should be to one side, and half to the other, but that is not what the funnel plot looks at.

    The pinnacle of the funnel indicates an estimate of the 'true' effect size so it could be found at any relative effect, depending on what the true effect size is. Therefore, as long as the funnel is roughly symmetrical about that effect size we can't necessarily ascribe publication bias.

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    pj: In general all a funnel plot needs to be, to show lack of bias, is symmetrical. But in this case it should be symmetrical around OR=1.00, because homeopathy cannot possibly work.

    If it were anything else – acupuncture, chiro, even intercessionary prayer – I'd be wary of ruling out a true effect. But I am confident that homeopathy is just water. If it isn't, we might as well give up the rest of science because it's all based on physics & chemistry, and that's all bunk.

    As for what a funnel plot would look like without any bias, it would depend of course on what the true effect was; it might well be asymmetric in this case because the trials are heterogeneous in terms of treatments and diseases (maybe good treatments are variable and bad ones are consistently bad). My point is just that the homeopathy funnel plot shows that you can get an apparently quite convincing funnel plot out of nothing at all.

  • http://www.blogger.com/profile/06832177812057826894 pj

    To say the funnel plot must be symmetrical about 1.0 is to beg the question – i.e. you have to have already assumed that homeopathy doesn't work – not an unreasonable assumption I grant you, but then you can't claim to be showing that there is bias with the plot because you've already assumed it.

    The point that you can get a fairly reasonable funnel plot with homeopathy is an interesting one, it perhaps highlights the general lack of sensitivity of funnel plots (which are far from specific either) – personally I'm not massively keen on them.

    One point worth making is that homeopathy trials are generally smaller than mainstream medical trials so that plot is an atypical sample. Funnily enough the mainstream plot shows more of the 'hollow plot' effect (i.e. borderline or non-significant trials being under-represented) than the homeopathy plot.

    A point that I think you do fail to make is that, despite that plot, heomapthy advocates pretty much all the homeopathic treatments available, but in medicine the results of RCTs goes on to actually influence what medications are and are not used – i.e. you don't need to be worried by the conventional medicine plot because it only tells you about the sum of medical treatments being researched not the treatmemts being used in routine practice.

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    “To say the funnel plot must be symmetrical about 1.0 is to beg the question – i.e. you have to have already assumed that homeopathy doesn't work – not an unreasonable assumption I grant you, but then you can't claim to be showing that there is bias with the plot because you've already assumed it.”

    Ah, but this is my whole point. There are limits to evidenced-based medicine and homeopathy is one of them; if the clinical evidence says that homeopathy works, i.e. that physics and chemistry are bunk, there must be something wrong with the clinical evidence.

    I grant that not everyone agrees with that perspective, some people would say that RCTs trump “preclinical” considerations… but I don't. Homeopathy is therefore a useful test case for EBM: if your approach to EBM concludes, from the published evidence, that homeopathy is better than placebo… you should rethink your approach to EBM.

  • Anonymous

    Anon 18:23 here:

    “Ah, but this is my whole point. There are limits to evidenced-based medicine and homeopathy is one of them; if the clinical evidence says that homeopathy works, i.e. that physics and chemistry are bunk, there must be something wrong with the clinical evidence.

    That homeopathy works (if it does) does not mean that physics and chemistry is bunk. It means that homeopathy works through laws outside of chemistry and physics. Simularity in plots also tells us that a lot of medicines we believe work through chemistry and physics in fact work through different mechanisms.

    I'm pretty sure anti-biotics work for pnemonia through quite biological laws. I think a lot of other areas are way greyer.

    And I completely agree that current EMB approach should be rethought. There are no RCT studies on effectiveness of parachutes yet they are still widely used and save lifes.

  • http://www.blogger.com/profile/06033188696919736448 Net Admin

    Evidence based medicine needs large trials and much repetition. Most of these trials referenced would not qualify as evidence base because they are too small. The only homeopathic trial that would be evidence based with enough patients would be the cochrane database oscillo trials. These trials should homeopathy has no effect on preventing the flu but may potentially less then flu by 6 hours. However, replication of this trial is necessary to make it real evidence. So people that say homeopathy works or does not work, no one really has enough prove either way.

  • http://www.cwars.org John Macgregor

    I'm not pushing the homeopathy barrow – I tend to be skeptical – but another explanation for your anomalous result could be that homeopathy does work – but by an as-yet-unestablished mechanism.

    Brian Josephson, co-winner of the Physics Nobel, hypothesises that there could be a 'sub-quantum domain' – something smaller than quantum – wherein lie the means for strange and wonderful effects to manifest, such as some of those in quantum, as well as various 'magical' phenomena…such as homeopathy.

    Just a stab, and very speculative.

  • http://www.blogger.com/profile/09949805075299293022 hakan altan

    thank you

  • http://www.encontactos.com www.encontactos.com

    It cannot have effect in reality, that is what I consider.

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Collapse bottom bar
+