Psychiatrists From Another Dimension (Part 2)

By Neuroskeptic | January 25, 2014 1:26 pm

In Part 1 of this post, I covered an emerging story of conflicts of interest within the American Psychiatric Association (APA). The controversy concerns a new “Computerized Adaptive Test” (CAT) that can be used to tell the severity of depression – a ‘dimensional’ measure.

I said that Part 2 would look at the test itself. But I’ve decided to split this further. In this post, I’ll be looking at the ‘practical’ aspects of the CAT. In Part 3 I’ll examine the science and statistics behind it.

To recap, the CAT is a software program developed by University of Chicago statistician Robert Gibbons, with the help of colleagues including David Kupfer, who headed the development of the DSM-5 manual. (N.B. I am here using “CAT” to refer to the CAT-DI – Depression Inventory. Gibbons et al have a family of other CATs for other mental health symptoms, at different stages of development.)

The CAT is essentially a self-report questionnaire – it estimates the severity of depression by asking people how they feel. However, unlike a simple pen and paper system, the CAT adaptively chooses which questions to ask, based on the subject’s responses to previous ones. There’s a bank of hundreds of questions, but any given subject only has to answer some 12. In a paper announcing the results of pilot studies, Gibbons et al say this provides for quick and accurate measurement.

How will this work in practice? This is unclear at present. Gibbons has formed a company, Psychiatric Assessment Inc. (also known as Adaptive Testing Technologies) and has issued founder’s shares to Kupfer, amongst others. Their website describes the CAT, but doesn’t describe how to get access to it, and doesn’t mention prices at all. Nonetheless, the fact that a company has been formed, and shares issued, suggests that profit is on the table.

If so, this might be a problem.

My fundamental concern is that the CAT could end up being closed-source; a ‘black box’. The questions that the patient answers are just the front end. The core of the system are the algorithms that decide which questions to ask, and then calculate the score, which would be displayed to the patient or their doctor.

blackboxVarious published papers have outlined how the CAT works, but (as far as I can see) the key details are missing – the full item bank and the various parameters, derived from the pilot studies, that determine how each question is handled.) In other words, no-one can go off and program their own replication of the CAT. And if someone wants to check whether the CAT has any bugs, say, they can’t.

A conventional questionnaire by contrast is (by its nature) open source. If there’s a misprint, you can see it. If there’s a question that doesn’t make sense in your context, you can delete it. You can study, research, and modify to your satisfaction. Copyright prevents you from publishing your own modification of many questionnaires, but you could still use them. In other words, with an old-fashioned questionnaire, you know what you’re getting, and if you don’t like it, you can change it..

The black box, ‘secret formula’ approach that CAT appears to be heading towards is problematic – but by no means unprecedented. Neuroskeptic readers may remember CNS Response and their EEG-based depression assessment, and the MDDScore blood test for depression – to name just two. Both of these rely on secret equations.

The oldest and by far the most successful of this genre is not from psychiatry at all. The Bispectral Index can be used to monitor the depth of anaesthesia. You hook it up to the patient’s head (it’s literally a box, although not always a black one) and it uses a secret algorithm to judge their state of consciousness based on their brain activity.

All of these cases have common problems from the perspective of you, the doctor using them (and by extension, the patients):

  • You can’t be sure how well the technology works and what its limitations are. You just have to trust the manufacturers – who of course, have a conflict of interest.
  • User innovation is impossible. There might be an easy way to improve the system or make it better suit your needs – but you can’t.
  • You’re paying money purely for the right to do something, not for the ability to do it (the hardware involved in all of the cases I mentioned is simple. If it wasn’t for the secret algorithms, it would be possible to implement these tests at low or zero cost.)

On this last point, you might object: doesn’t an inventor have a right to make money from his or her invention? In a free market, shouldn’t people be able to market the fruits of their labor?

Perhaps, but the CAT is no product of capitalism: it was developed using public money. Robert Gibbons has received $4,958,346 in National Institutes of Health (NIH) grants since 2002. The project title: Computerized Adaptive Testing – Depression Inventory. Robert Gibbons is no John Galt.

Maybe I’m jumping the gun here. No-one is monetizing the CAT yet… but if someone does, then the NIH would effectively have been providing start-up funds for a commercial enterprise. Eventually, CAT might become available on Medicare or Medicaid, in which case the American taxpayer would, outrageously, be paying for the privilege of using a product that they paid for in the first place.

But this hasn’t happened yet. Perhaps Psychiatric Assessment Inc. will turn into a nonprofit and the CAT will end up being free. How useful would it be? Find out in Part 3.

  • Bernard Carroll

    There are 2 black boxes – the one you described and also the funding and business plan of this start-up corporation. Further discussion is over on Health Care Renewal today:

  • Uncle Al

    paying for the privilege of using a product that they paid for in the first place.” Obamacare is paying for the privilege of not using a product that they paid for in the first place. Depression is a sane reaction to living in a toxic society. Depression is anger without enthusiasm.

    • Nitric-X

      I think comments like these are really OT – anybody who has ever experienced clinical depression, or worked with patients suffering thereof, will never say that depression is a “sane reaction”. This comment, sorry to say that, is complete bulls***.

    • Jon Nixon

      Well, that solves it. Prozac and it’s cousins are not anti-depressives, they are pro- enthusiasm drugs. I knew that all along……..

  • cannotsay

    I find this fabricated outrage quite comical.

    And since I like humor a lot, I thought I would illustrate the issue with a
    joke about San Francisco that was very popular among conservatives. I do not find the internet reference, so it must have lost some popularity. I have to retell it using my own words, which might not be as funny.

    The joke was something like this. It talked about a party attended by
    the mayor and the San Francisco elite. The party was full of the usual
    negatives about San Francisco that have become anathema in the
    conservative consciousness: wild alcohol consumption, heavy illegal drug usage (cocaine, crack, heroin, etc), sessions of group sex, old gay men having sex with minors, etc. Suddenly the mayor decides to pull a stunt in the middle of the room with a dominatrix and a bottle of Vodka with everybody watching. The mayor is unlucky and the bottle falls to the floor where it is left for the remaining of the party.

    The next day, the San Francisco Chronicle, upon learning about the
    party, decides to take the story to its front page in a responsible
    exercise of investigative journalism. The headline “San Franciscans
    express outrage that their mayor doesn’t recycle bottles of Vodka in his parties”.

    It must be an “insider thing”, but seriously worrying about this when
    people are dying everyday of the effects of the drugs that psychiatrists prescribe to treat psychiatry’s invented diseases seems
    like a practical joke that only psychiatrists can understand :D.

  • petrossa

    funny how black boxes are at work in the field. fMRI is a very prominent one but not really recognized as such. Which is strange

    • Neuroskeptic

      fMRI is not a black box; fMRI physicists and statisticians do understand how it works.

      However, it’s true that many neuroscientists do use it as a black box.

      • petrossa

        your correction is correct :) I should have stated it more clearly. Glad you rephrased it better

  • Neuroskeptic

    Generic antipsychiatry rants will be met with generic antirant deletions.

    Please keep the discussion on the topic of black boxes, CAT and the Gibbons-Kupfer Gang.

    • cannotsay


      I had my comment removed. I came here because Sandra Steingard mentioned your blog entry in the comments section of her own piece

      Now, if you read the comments there, most had the same thing to say as what I said. The last comment is very graphic “two different gangs of drug dealers are arguing over spitting up the profits”.

      Honestly, with all the things that are wrong with psychiatry, including in the area of conflict of interests -like psychiatrists topping the lists of Big Pharma payments to doctors- to have an outrage about this feels like a practical joke for many of us.

      I for one, feel more outrage about this than the minuscule manufactured scandal that you raise here.

      • Neuroskeptic

        This comment can stand because it does relate to the topic – however, it is asinine.

        I’m not manufacturing any outrage and if you look back over my posts (e.g. the “antidepressants” and “1 in 4” tags) you’d see that over-prescription of medication is one of my regular topics.

        I’ve been criticizing bad psychiatry for years; here’s three examples of times when my points were, later, taken up in the psychiatry journals:

        I am also, believe it or not, a “victim” of psychiatry myself, by which I mean I am currently taking a number of psych meds. So please understand that when I reject your point of view it is not out of ignorance.

        • cannotsay

          I wasn’t familiar with this website, so I apologize.

          Still I feel that Mr Carroll and Mickey Nardo are manufacturers of this crisis. They have not experienced psychiatric drugging on their own bodies; they just made a living out of forcing drugs on people. Even Sandra Steingard feels in her replies above that the subject is likely to be uninteresting to the people, like yours truly, whose major beef with psychiatry is having been involuntarily committed for several weeks (in a European country with “need for treatment” standard) and drugged against his will for several months in the same country.

          So from I stand, this COI is like San Franciscans complaining about that bottle of Vodka not being recycled when the larger issues in psychiatry are much worse. A distraction to take the conversation where believers in psychiatry like Carroll and Nardo want to take it.

  • Nitric-X

    I have not yet seen the test, so this comment is rather preliminary. However. There are 2 sides to this, a scientific one and a clinical one. As for the scientific side – rating depression severity for pharma trials or whatever study – there are gazillions of free and validated tests out there and I don’t see the reason for yet another one. From the clincal perspective… well, we all know that rating scales can never capture the entire subjective suffering of the patient. Patients with the same BDI score can have quite different MADRS scores (and vice versa), as the first one is self-rated; and people with either the same BDI or MADRS can suffer quite differently, and so on. Do we need another scale to rate this? IMHO no – seriously, if you can’t assess depression severity in a thorough clinical interview, you probably should think about repairing cars instead. This pseudo-objectivity and hence over-estimation of rating scales completely annoys me; these are core issues of your psychiatric skills, and any rating scale or related measure can always only be something in addition. Seeing the decline of clincal knowledge in US psychiatry (e.g. I’ve heard a dozen of times “our doctors don’t prescribe lithium, as it is too complicated” – come on! Introducing a 4-drug combo of chemotherapy for glioblastome is complicated, but not lithium!), the APA should rather focus on thorough education instead of introducing unneccessary instruments like these.

  • Robert T. Rubin, MD, PhD

    One issue that apparently hasn’t yet been mentioned is that of petitio principii: Taking as a given truth the premise that a self-report of symptoms by a depressed person will accurately and completely capture that person’s totality of symptoms and signs, without omission, exaggeration, or minimization of any one symptom. Based on that unproven premise, deriving subsequent questions from the initial answers could lead to magnification of errors and highly inaccurate conclusions. Clinical trials of any rating scale, no matter how derived or mathematically sophisticated, must be done to indicate the real-world applicability of the new test compared to current psychometric and clinical standards.

    Robert T. Rubin, MD, PhD

    • cannotsay

      You are venturing into “anti psychiatry waters”. The idea the none of the DSM labels can be objectively verified by biological tests is the broader question in psychiatry that your field has been wrestling with since the Rosenhan experiment. So this questionnaire that Kupfer wants to make the gold standard of psychiatry vs the questionnaire that Bernard Carroll pushed is a very minor issue in the larger scheme of things.

  • Robert T. Rubin, MD, PhD

    Dear Cannotsay (and all the other non-physicians reading this blog):

    Ironic your saying that I’m venturing into anti-psychiatry waters, since I am a senior professor of psychiatry at UCLA and Distinguished Life Fellow of the American Psychiatric Association. Who else better to offer considered criticism?

    Which highlights that your phrase “anti-psychiatry waters” is an artifice, a fiction; just as pro-psychiatry is a fiction. Are you anti-endocrinology or pro-endocrinology? Anti-orthopedics or pro-orthopedics? Makes no sense. What is important is that every medical specialty, including psychiatry, is grounded in the best scientific database that we have at the time (and will continue to refine). That includes the spectrum from randomized, double-blind, clinical trials to clinicians’ experiences with individual patients, all packaged with a large measure of treating one’s patient as the physician him/herself would wish to be treated. For something current as of this morning and profoundly enlightening, read:

    So let’s think clearly about what is good and what needs improving in all of medicine, not just psychiatry. Regards, RTR

    • cannotsay

      I think that there has been a misunderstanding here. My reference to “anti psychiatry waters” is because neuroskeptic has said he will delete all anti psychiatry discussions that are not related to the matter at hand.

      I am a proud antipsychiatry activist who will challenge anybody, including you, to prove me through biological objective testing that does not involve a subjective determination by a psychiatrist that one, only one, of the DSM labels is as real as HIV is real or CJD is real. And I stop here because I don’t want my comment removed.

      And I add, because it is relevant, that Bernard Carroll bailed out of a similar debate at Phil HIckey’s debate when he was miserably losing on the intellectual merits of the proposition.

      • Neuroskeptic

        Anyone taking you up on your challenge will find their comment removed.

        • cannotsay

          Which is why I made a brief mention and decided to shut up. Bernard Carroll lost last time though :D.

      • Neuroskeptic

        I’m only leaving this comment here because of the (useful) clarification in the first half.

  • Pingback: Psychiatrists From Another Dimension (Part 2) | Health Tips - Your Health Team | Health Tips - Your Health Team()

  • Nick Stuart

    What a strange thread for me… Uncle Al – how did the Eötvös experiment work out?.. I used to follow your threads on science a few years back. Cannotsay… nice to see how you are still being censored.. you must be saying something interesting.
    Stanley Milgram demonstrated years ago how obedience to white coats can affect outcomes especially when using HAM-D type rubbish! (Anyone here ever read Ham-d?) And neuroskeptic… I am still the only Szaszian in the village! Ha! Of course I am anti-psychiatry in the same way I am pro-science. So NS, feel free to delete this comment…. ;-(

    • cannotsay

      Hi Nick,

      It will not take long before your comment and my own will be deleted :D. In the meanwhile, nice to see you here as well ; It seems this guy is one of your fellow British citizens 😀 .

  • Nick Stuart

    On self reporting scales….like the HAM-D

    Psych. ”How do you feel?’
    Patient. ‘Depressed’
    After a few ECT sessions and a chemical lobotomy..
    Psych. “How do you feel after all this we have done for you to make you feel better?
    Patient. Much better thank you.

    Medical Science? Or what? 😉

    Of course it is well known that the benefits of ECT are deemed ‘good’ when explained by the patient to the doctor than when described to a bystander… I can did out the paper if you want…



  • Pingback: Can A Computer Measure Your Mood? (CAT Part 3) - Neuroskeptic |



No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.


See More

@Neuro_Skeptic on Twitter


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Collapse bottom bar