Can Psychologists Learn More by Studying Fewer People?

By Neuroskeptic | July 10, 2016 8:25 am

In a brief new Frontiers in Psychology paper, Matthew P. Normand argues that Less Is More: Psychologists Can Learn More by Studying Fewer People.

crowd_psychology

Normand writes that the conventional wisdom – that a bigger sample size is better – is wrong. Repeated measurements of a few subjects, or even just one individual, can be more informative than casting the net widely, he says

Psychologists tend to view the population of interest to be people, with the number of individuals studied taking precedent over the extent to which each individual is studied. Unfortunately, studying large groups of people makes repeated measurement of any one person difficult. The consequence is that we often end up knowing very little about very many…

When you have only one or a few measures of each individual’s performance, it is impossible to know how representative those measures are for the individual, never mind the population.

He goes on to outline the advantages of single case designs

Single-case designs involve the intensive study of individual subjects using repeated measures of performance, with each subject exposed to the independent variable(s) and each subject serving as their own control… Comparisons of performance under baseline and experimental conditions are made for each subject, with any experimental effects replicated with the individual subject across time or across multiple subjects in the same experiment. Single-case experiments yield data that can be interpreted using non-inferential statistics and visual analysis of graphed data, a strategy characteristic of other natural sciences.

Normand argues that because individual psychological traits and performance on tasks are variable over time, we need to measure the same individuals repeatedly over time to capture the psychological reality that is within-subject variability

Variability is something to be understood, not ignored. To average it away is to assume that it is unimportant because it does not represent the real world. But variability does not obscure the real world, it is the real world.

He acknowledges that the representativeness of the subject(s) in a small-n study might be a concern, but he says that

Despite the advantages in terms of internal validity, some assume that findings from single-case designs have limited external validity because data obtained from a few subjects might not generalize to a population at large.

Actually… Generality is best established inductively, moving from the single case to ever-larger collections of single cases experiments with high internal validity. To have external validity you must first have internal validity.

Hmm. Normand makes a good case that small, intensive studies are a legitimate research strategy in psychology. He doesn’t showcase any examples where this strategy has led to new insights, however, which leaves Normand’s paper open to the charge of being “hand-wavey”.

In neuroscience, single-case studies are quite common when it comes to individuals with rare brain lesions, such as the famous case Patient H.M. But these single patient studies are something of an exception. In the rest of neuroscience, there’s generally a desire to maximize the sample size, although there has been at least one single subject repeated neuroimaging study.

ResearchBlogging.orgNormand, M. (2016). Less Is More: Psychologists Can Learn More by Studying Fewer People Frontiers in Psychology, 7 DOI: 10.3389/fpsyg.2016.00934

CATEGORIZED UNDER: methods, papers, select, Top Posts
ADVERTISEMENT
  • http://www.mazepath.com/uncleal/qz4.htm Uncle Al

    WWII required hugely more US psychiatrists than existed, and could not tolerate four years of analysis, hence the “psychologist.” A psychologist recycled city boys crushed by confrères being slaughtered. Shut ’em up and put ’em back, conveying the appearance of healing. The US otherwise ignores used soldiers unless they kill false authorities.

    Psychology is a bone without a dog. Any statement is an Officially valid statement. “Woof woof.”

  • Dr. Turi

    Try Astropsychology – Astroforensics and get all the answers…. 66 years of research, simply unmatched! Google “Dr. Turi” for more…

  • Matt Normand

    Thank you for the comments on my paper. You ask a good question about what has been found using single-case designs. The “Opinion” category for Frontiers allows only 2,000 words, so I was limited in the detail I could provide.

    In brief, and for the field of psychology, I would highlight the work of Ivan Pavlov, one of the most important scientists of the 19th Century, and B. F. Skinner, arguably the most influential psychologist of the 20th Century. Both relied on single-case designs, which are not the same as case studies (like Patient H-M). Single-case designs are experimental designs that demonstrate functional relationships, not just correlations.

    You can find many examples in the Journal of the Experimental Analysis of Behavior (http://onlinelibrary.wiley.com/journal/10.1002/(ISSN)1938-3711) and in the Journal of Applied Behavior Analysis (http://onlinelibrary.wiley.com/journal/10.1002/(ISSN)1938-3703), to name just two sources.

    Thanks again for your comments.

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      Many thanks for stopping by! You’re quite right, Pavlov is a good example of the single-case design (I had forgotten that).

  • David Max

    Normand’s point is well taken. In fact, before the logical positivists and statisticians carried out their coup of research design in psychology (and the other social sciences as well as other disciplines), all of the research in psychology was carried out using within-subject designs, including that by Wundt, Ebbinghaus, Pfungst, etc. And, as Normans mentions, Pavlov and Skinner’s research–culminating in some of the only natural laws of behavior ever discovered–used within-subject designs exclusively. One problem is the monicker, “single-case” because of the inclusion of the term “case.” This can be avoided by referring to such designs as “within-subject” or “single-subject,” although the latter may give the wrong impression that only one subject is ever used. Either way, there is a place in psychology for both within- and between subject designs, but unfortunately all most psychology students ever learn about and how to do are between- subject designs which, as Normand notes in his article, only serves to obscure variability–the spice of life.

    • tim faber

      I would say it depends on the field, cognitive psychology predominantly uses studies with within subjects designs.

  • polistra24

    Well, “social” “science” conclusions are written by the voices in the researcher’s head anyway. Might as well save money and go for zero subjects.

    • Kez

      This ^
      …….well in the social theory, ideological end

  • David Palmer

    I agree with Normand. The principles of behavior operate at the level of the individual organism and should be studied at that level. Generality can be assessed by replicating with other individual organisms, as often as needed.

    • OWilson

      I’m no expert, but there is such a phenomena as ‘crowd behavior” which can apparently override the individual’s normal proclivities.

      Two separate areas of study?

  • Vaughan

    “these single patient studies are something of an exception”

    Although,not such an exception that there aren’t 100s of published studies and whole journals explicitly focused on them (Cognitive Neuropsychology, Neurocase).

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      I stand corrected!

      • kndiaye

        Really interesting paper, Matt — and great comments here as usual! I would also add the whole field of psychophysics where N of 2 to 5 are quite common. Granted, this is in part due to experimental designs where the number of trials to collect may be in the 1000s per subject (e.g. doi:10.1016/S0042-6989(02)00019-6 or recently in Neuron: doi:10.1016/j.neuron.2016.03.025) Also in primate comparative psychology (and neurophysiology) N is usual below 5
        Finally, one could cast this trend into a more global change of perspective akin to the so-called N-of-1 approach in medicine: http://www.nature.com/news/personalized-medicine-time-for-one-person-trials-1.17411

    • Matt Normand

      Those journals are outside of my field, so I can’t comment on them specifically. However, I do want to again raise the distinction between case studies (which are correlational) and single-case experimental (emphasis on experimental) designs, which reveal functional relations. (And, of course, single-case designs can involve many subjects–but experimental control is demonstrated with each individual subject.) The journals you cite might, indeed, publish single-case experiments, not case studies, but I don’t know…

  • Niall Conlon

    I recently read this article on the use of single-case designs in early intervention/early childhood special education that might be of interest to some:

    http://tec.sagepub.com/content/36/1/4.full.pdf+html

    One of the advantages of SCEDs is that they can be used relatively easily in practice. Given that most psychology students go on to work in practical settings, it is strange that so many undergrad courses do not include SCEDs as part of the research methods they teach.

  • Anonymouse

    I agree that within-subject variability is a concern and that increasing internal validity by studying individuals more thoroughly is a worthwhile thing to do, even though I’m not too convinced by that opinion peace, especially since it makes some bold claims about how psychology should go about its business and what the world is really like that are at least debateable.

    I have to say that I am somewhat biased, because I am in a field where within-subject repeated measures designs are fairly common and time-series are often not averaged to a single number (psycholinguistics). Comparing this to between-subject single-measure group comparisons therefore isn’t really interesting to me and I am probably less angry about how experiments are designed and carried out, however obviously the superiority of within-subject repeated measures is in terms of power and the kind of information it can provide, wherever it’s possible. However, the main point of this opinion peace is to look more and first at individual subjects and less at averages in general.

    That the results averaged over large sample sizes can be questioned is a criticism I’m familiar with. The head of the lab I work at – a fairly big figure in our field who probably knows his statistics better than I ever will – is also very unhappy with relating averages to averages, especially for very messy data and complex models. His reasoning is that some of the effects in, say, EEG that you get from partialling out everything else don’t seem to ever be observable in any single individual. So an N400 might look like an actual negativity in a couple of the subjects (though with high varaince in its amplitude), but most of them don’t really show any such trend (observable when looking at the actual curves), with some even showing the exact opposite pattern.

    This is a difficult issue.
    On the one hand, you would like the effects that your analysis reveals to be visible in the raw data. If you’re looking at temperature over a year, a seasonal trend should be observable in most years that you look at individually. After all the mean that statistical models fit in one way or another should tell you where most of the data points cluster, but if none of the idividual data points are anywhere near this mean, how meaningful is it really?
    On the other hand, our world is messy and statistical models should help reveal those effects that we can’t observe, too, maybe even especially those. From what I can tell, the people who are most familiar with statistics tend to also those who are most skeptical of this, thoug. So when I just talked about this to the head of my lab, he quoted Douglas Bates saying that if he can’t actually see an effect, he doesn’t trust it.

    I have no clear opinion on this statistical issue. What I do want to stress, though, is that looking at any individual is clearly limited in what it can in principle tell you, statistical issues aside. For example, basic learning theory tells us that learning doesn’t happen in a vacuum, meaning that what a subject learns in a given experimental manipulation depends on everything that subject has already learned. What that means is that I can’t just give a someone something to learn and observe what happens, because, different from Pavlov’s dog or Skinner’s pigeons, I have no way of controlling what said individual already knows, which, in the worst case, renders my observations anything but uninformative. Collecting a bunch of those detailed single-case observations and then trying to find their similarities, if at all possible, seems like a massive waste of resources. And my feeling is that this will be the case in general where observations from several people will be highly mutually uninformative.

    A shift from between-subject single-measure to within-subject
    repeated-measure designs is obviously a good thing in general. Single-case designs can be very useful, definitely, but I disagree they are always the way to go or the way to go first. And experimentally “controlling” everything that is possible is desirable, but so is “controlling for” everything that is out of the experiments’ reach. I don’t see how any of these do away with inferential statistics though.

  • zlop

    Steve Pieczenik studied the 2 important ones.

    “Steve Pieczenik, who said that not only was President Obama gay, but he also has a romantic past with Chicago mayor Rahm Emanuel.”

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      A rigorous n-of-2 study I’m sure

      • zlop

        I saw sporting images. Supposedly, Emmanuel bosses Obama around. In depth study gives an edge in international relationships. Obama tucked Cameron into bed. What of Theresa May? . .. …

  • Pingback: Weekend reads: More Impact Factor scrutiny; $10 million fine for overbilling; protected Canadian fraudsters - Retraction Watch at Retraction Watch()

  • D Samuel Schwarzkopf

    Interesting! Unsurprisingly (given that I study within-individual variability in my own research), I agree to some extent. But it’s not that black and white. Surely this depends a lot on the research question. If you want to establish some common factor about the human mind/brain you do need to cast a wider net. The findings that even basic perceptual processing might vary between different populations shows that case (or small n) studies could be potentially quite misleading. And for some research fields, like social psychology, it is probably also doubtful that you could just repeat this experiment over and over (even though I do think social psychologists would be well advised to at least try to change their approach somewhat – such as coming up with within-subject designs of their experiments).

  • Pingback: Google()

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Collapse bottom bar
+