How Well Do Drugs Work? Hidden Research Sometimes Makes It Hard to Tell

By Veronique Greenwood | January 6, 2012 2:14 pm

What’s the News: Scientific publishing is how researchers get the word out about what they’ve learned. It’s how people outside the lab learn whether a drug was safe or not, or whether a treatment had any effect. But a team of scientists looking at a sample of clinical trials funded by the US government found that 30 months after the trials were completed, more than half had not yet been published. And that means that other studies trying to assess whether those treatments are safe and effective are working with incomplete information, while the relevant trials, already paid for by the public, are gathering dust on a lab bench.

How the Heck:

  • The team, whose study appears in a special report in the British Medical Journal, looked at clinical trials listed as having finished in December 2008 on
  • Of the 635 they examined, 341 (54%) were not published in a peer-reviewed journal within 30 months of the end of the trials. 51 months after completion—that’s over four years—a third of the trials remained unpublished.
  • That could have a serious effect on research and health. Another team whose work appears in the BMJ special issue took information the FDA made available from unpublished trials on drugs it approved about ten years ago. They then reran meta-analyses looking at how effective drugs were, now with the previously unpublished results included.
  • They found, chillingly, that including the unpublished data could change the conclusions reached by meta-analyses. The researchers write: “Overall, addition of unpublished FDA trial data caused 46% (19/41) of the summary estimates from the meta-analyses to show lower efficacy of the drug, 7% (3/41) to show identical efficacy, and 46% (19/41) to show greater efficacy.” That means that when scientists try to study those FDA-approved drugs, they may not realize that they work less well than published papers indicate (or better, as the case may be).

What’s the Context:

  • This is far from the first time it’s been shown that trials go years and years without being published: previous studies have found that 25% to 50% of clinical trials are unpublished years after completion. But the first study mentioned above is the only one since 1979 looking at studies funded by the US National Institutes of Health, which doles out taxpayer dollars to support important research projects related to human health.
  • Part of the agreement you make when you receive a grant from the NIH is that your work must be listed on the PubMedCentral public database within twelve months of having been published. But there aren’t hard and fast requirements on when it has to get published.
  • There are a variety of reasons that trial results might take a long time to publish: maybe it’s not the biggest priority in the lab (or as a grad student once jokingly said to me, “I know the answer now. Who cares if anyone else does?”), maybe the researchers are having trouble finding a (sufficiently prestigious) journal to publish it in, or maybe the team is reluctant to share less than stellar results. And as far as releasing the information before it’s published is concerned, in biology such a thing is almost never done: A journal’s peer reviewers might spot some flaw in the analysis that the researchers hadn’t noticed, of course, but there are also numerous taboos against letting the cat out of the bag, related to problems of intellectual property and the climate of competition within the field.

The Future Holds:

  • It’s understandable that it can take some time to get scientific results together for publication. But these numbers are frankly frightening. Given that clinical trials are explicitly about advancing people’s health, it seems like there should be a premium placed on getting that information out there.
  • The arXiv, a database of yet-to-be-published physics and math papers maintained by Cornell, has become a useful resource for researchers, journalists, and the interested public. Several years ago, Nature set up Nature Preceedings, which acts as an arXiv for biology, and though it hasn’t really taken off, probably for the reasons mentioned above, maybe that model is what’s required for clinical trials. Perhaps a new requirement for accepting NIH funding should be publication in such a preprint database, if not a regular journal, before a certain number of years are up.

References: J. S. Ross, T. Tse, D. A. Zarin, H. Xu, L. Zhou, H. M. Krumholz. Publication of NIH funded trials registered in cross sectional analysis. BMJ, 2012; 344 (jan03 1): d7292 DOI: 10.1136/bmj.d7292

B. Hart, A. Lundh, L. Bero. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ, 2012; 344 doi: 10.1136/bmj.d7202.

Update: The headline on this story has been changed to make the central message clearer.

CATEGORIZED UNDER: Health & Medicine, Top Posts
  • drkala

    This seems like a trivial finding. If you increase the size of the sample in meta analysis, you increase the power of the finding. Therefore, you would expect to see stronger results in both directions–work better, work worse. That is a consequence of increasing the number of underlying data points, by adding more studies. What would be problematic would be if the unpublished findings were skewed one way, say not publishing ones that show they are working worse. Since the changes are equally distributed, this actually validates the publcation process and confirms that there has not been a publication bias.
    Kala Ladenheim, PhD, MSPH

  • Bob Snyder

    @drkala, My understanding of the article was that the issue is in regards to people making decisions with incomplete information and taxpayer dollars going into clinical trials which take years to get published after they are completed(because there is a lack of control), not whether or not “publication bias” exists.

  • dave chamberlin

    you mean big pharma has clout in obscuring what they don’t want known? naaaaa. They are in it just to help people.

  • Iain

    All investors are entitled to see what their money has done. If taxpayers are funding a study, it needs to be published in a public accessible data base within a reasonable time, say 12 months after study complete or refund monies received. Also the database needs to be advertised so people can easily find and access it. A single government spot with sub-directories should suffice.

  • TouchTheRiot

    We need to make it illegal to not publish the results within one year of the completion of the study and make them ineligible to even request more tax payer money with any outstanding study results unpublished. If tax money is what makes a finding possible, then the tax payers should be the owners of the IP, but the researcher can receive a sizable royalty. We need overwatch and accountability, geez.

  • derp gellington

    “Latest results on a few studies show our drug isn’t actually very effective.”
    “Slip some cash discretely to the lead scientists not to publish and no one will ever know. But make sure any favourable studies do get published. If the general public and doctors aren’t told this drug barely works we’ll make billions. So crush as many negative studies as you can. Who’s gonna stop us? The FDA? Ah hahahaha”

    Just another day in the office with our lovely, bloated, and corrupt big pharma. Money has always been more important then human lives to these people.


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!


80beats is DISCOVER's news aggregator, weaving together the choicest tidbits from the best articles covering the day's most compelling topics.

See More

Collapse bottom bar