How might we leverage knowing that a particular neurological feature makes someone more vulnerable to autism or Alzheimer’s or more likely to achieve academically?
In March, neuroscientists and psychiatrists from the School of Medicine at Washington University, St. Louis, along with colleagues elsewhere, published a study in the journal Nature that sparked widespread discussion in their fields. Researchers, the study noted, are increasingly using magnetic resonance imaging — which can reveal the brain’s structure and activity — to try to find links between what is seen on an M.R.I., like cortical thickness or patterns of connection, and complicated psychological traits, like cognitive ability or mental-health conditions. In theory, such so-called brain-wide association studies could yield incredibly valuable insights. Knowing that a particular neurological feature makes someone more vulnerable to autism, Alzheimer’s or another disorder, for example, could help predict, prevent or treat that condition. Likewise, if we can link certain features to desirable traits, like academic achievement, it might be possible to take advantage of that knowledge.
The problem, the Nature authors argued, is that neuroscientists often are searching for those associations in groups of study subjects that are too small, leading to results that are statistically “underpowered.” In general, they calculated, thousands of subjects should be included for a brain-wide association study to produce a finding that other studies can replicate. This was unwelcome news to many, in large part because M.R.I. machines are incredibly expensive to use, often at about $1,000 per hour, and funding is limited.
Credit…Illustration by Ori Toor
Specific instances of underpowered studies are legion. So much so, says Terry Jernigan, director of the Center for Human Development at the University of California, San Diego, that singling out an example “would simply be unfair.” Indeed, according to a paper from 2020 in NeuroImage, the average number of study subjects in more than a thousand of the most cited brain-imaging papers, published between 1990 and 2012, was 12; the Nature paper calculated that the median sample size for neuroimaging studies uploaded to a popular open-access platform as of September 2021 was 23.
Unfortunately, small M.R.I. samples frequently return strong associations as a matter of chance. For example, let’s say you want to see if there is a correlation between eye color and a preference for strawberries. If you look at enough groups of 25 random people, eventually you will happen upon a group in which blue-eyed people like strawberries much more than brown-eyed people do. But if five independent research groups run this study and only one of them finds this relationship between eye color and a love of berries, that will be the group most likely to publish its results — despite returning the least representative results. That’s because journals historically have preferred surprising correlations to findings of no correlation, a phenomenon known as publication bias. “The paradoxical effect is that the answer that’s the most wrong gets published if you use a small sample,” says Nico Dosenbach, an associate professor of neurology at Washington University and an author of the Nature study.
Scientists across disciplines have long known about this dynamic, but the Nature paper was able to pinpoint — at least in the case of brain-wide association studies — how many participants are needed to avoid it. Using M.R.I. data from about 50,000 people, the authors searched for links between brain structure or activity and complex psychological traits in groups with different numbers of subjects. Subjects had to number in the thousands, on average, for studies to be replicated reliably.
The fact that so many associational studies are underpowered — and often untested in other groups of subjects before publication — has led to reports of myriad links between brain features and psychiatric disorders that are probably unreliable. These can be frightening and stigmatizing. “If you see a particular brain-activation pattern in a person with a psychiatric diagnosis, that doesn’t mean it’s causing the disorder or symptoms,” Jernigan says. “It’s simply an association.”
But the conclusion of the Nature paper applies only to studies that compare M.R.I.s from multiple people in order to identify differences among them relating to complex mental traits. Neuroimaging studies that show brain changes taking place within individuals, on the other hand, can be dependable even with very few participants. For instance, the first notable paper to demonstrate that most people’s brains work in roughly the same way appeared in Science in 2001 and included only six participants, says Russell A. Poldrack, a professor of psychology at Stanford University. That study’s researchers recorded each subject’s brain activity while viewing pictures of cats, faces, man-made objects and nonsense images. It didn’t matter that each brain was unique — the changes that took place in that brain could be assigned to seeing different types of pictures. The patterns were then tested and found to correctly predict, based on brain activity, what a participant was seeing. Those overall patterns, along with other evidence, Poldrack says, established that “when people engage in particular kinds of mental functions, particular brain areas become engaged.”
This realization that we tend to share brain patterns raises the tantalizing possibility that somewhere in the variations among them lies an explanation for why some people have a particular trait or collection of symptoms that others lack. But it’s extremely difficult to separate meaningful differences from the countless random differences that exist between all brains. One way to try to do so is to compare the M.R.I.s of thousands of people and look for a variation — a certain pattern of neural connectivity, say — that is more common in those with a particular psychological condition. Recent advances in M.R.I. technology, and in the ability to analyze vast amounts of data, have begun to make this sort of effort possible. For example, the Adolescent Brain Cognitive Development study has enrolled nearly 12,000 children in the United States between the ages of 9 and 10 whose brains will be scanned regularly into young adulthood. The study will also track socioeconomic variables, like parental income, and psychological attributes, like resilience, to see how they intertwine with brain development. “Without a study like this, you could never resolve these questions,” says Jernigan, a director of the study’s coordinating center.
Such comprehensive data sets can enable researchers to interrogate the relationships between many factors at once. But there are other ways to explore the connections between brain features and complex mental functions that don’t require as many subjects. Rather than look for associations across a whole population, you could start out by doing “a very detailed characterization” of the brains of hundreds of people diagnosed with various types of depression, for instance, to build models of what those states look like, says Roberta Diaz Brinton, director of the Center for Innovation in Brain Science at the University of Arizona College of Medicine.
In a paper from 2016, Monica Rosenberg and her colleagues at Yale University used this type of approach to identify certain kinds of brain activity in someone paying sustained attention. They gave 25 healthy adult volunteers a task that required focus and made a composite map of what they saw happening in the brains. Then they compared that map with M.R.I. data from 113 children with and without A.D.H.D. The sustained attention network they had identified was weaker in the children with A.D.H.D., which affirmed their result.
Rosenberg, now a professor of psychology at the University of Chicago, reads the Nature paper not simply as a call for larger sample sizes but as a broader reminder of the need to “systematically follow up on results.” For any finding, she says, scientists should ask: “Is this true in a different group of people? Individuals of a different age? How general is this relationship?” Though she says there is still “significant room for improvement,” Rosenberg is optimistic that the Nature paper is one of many in recent years that push for better practices in brain-imaging research. “The value of replication is increasingly being recognized,” Rosenberg says.
Kim Tingley is a contributing writer for the magazine.