You’ll often see loony zealots refer you to a study showing how effective their preferred treatment is — there usually is some small study supporting the use of almost any treatment.
You’ll also often hear people reply that the study was only small, so shouldn’t be trusted. But why shouldn’t you trust small studies? Sure, they won’t provide quite as much statistical power as larger ones, but surely they can still be useful.
And that’s true. They can be useful, and they do provide important information. But a meta-epidemiological study in the British Medical Journal recently showed a really interesting fact about small studies.
The researchers highlight what is known as the “small study effect”: a very particular bias that small studies introduce into systematic reviews.
It turns out that small studies are systematically biassed towards the effectiveness of the intervention they are testing.
Systematic reviews pool the results of all the relevant studies on a particular issue and usually provide the very best evidence. Before major decisions are made about some particular treatment, we usually wait for a big systematic review of the literature to be published.
But if there have been lots of small studies done, then when researchers conduct a systematic review, it turns out they might end up being slanted.
These particular researchers looked at studies that tested various treatments for osteoarthritis and they plotted all the studies for each treatment on a graph with the larger trials near the top and the smallest ones near the bottom.
If the study showed the treatment was very effective, they plotted it further to the left and if it showed it was ineffective (or had a negative effect) they plotted it further the the right.
They call them funnel plots because if small trials are not biassed, the graphs will resemble funnels. The large studies will group together at the top and the small studies will scatter evenly on either side.
The results are visually striking. Far from resembling funnels, they resemble toppling towers — with small studies heavily drawing the plots to the left, towards the treatment being more effective.
The funnel plots for the 13 meta-analyses studied.
You can see just by looking at these plots that if systematic reviews are conducted, pooling all these data, the final analysis will generally be skewed towards supporting the treatment.
So why are small studies biassed in this way? The study authors suggest that there might be several factors at play. For one thing, there might be a selection bias: Small studies that show less effect might be less likely to be published. They also suggest a number of other explanations including a problem with participants being excluded from the analysis after being randomised into one of the arms.
They urge that authors of systematic reviews include the above funnel plots in all systematic reviews, and if a “small study effect” is observed, the reviewers should include a separate analysis that excludes all the small studies.
Update: The researchers defined “small” studies in this paper as ones with less than an average of 100 participants in each arm. Thanks, Simon, for pointing this out in the comments.
So next time someone points you to a small study showing how effective acupuncture is, how reflexology relieves depression or how fish oil cures everything, you can rest easy knowing that you’re under no obligation to accept the study’s conclusions. It’s much better to wait for a larger study, a meta-analysis, or even better, a meta-analysis that controls for the small study effect.
Nüesch E, Trelle S, Reichenbach S, Rutjes AW, Tschannen B, Altman DG, Egger M, & Jüni P (2010). Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study. BMJ, 341 PMID: 20639294
When talking about ’small studies’ it’s probably useful to describe the kind of sample size you’re talking about. From the paper, we can see that they classified a study as ‘large’ if the average number of patients in a treatment arm was 100, for example one with 110 in one arm and 95 in another would be ‘Large’ because the average number of patients is 102.5. This is crucial information for someone trying to understand your blog post and its relevance to their particular interest.
Yes — You’re absolutely right. In fact I did say it in the first version I wrote and must have accidentally deleted it in an edit. Thanks!
Actually, just a further thought: Their definition of what a small study amounts to is not so crucial for understanding the post since the analysis I focused on was the funnel plots. In them, the y-axis is (something similar to) sample size and they do not merely compare “small” studies with non-small ones.
Nevertheless, it was remiss to not mention their definition. Thanks.
Yes, I realised upon reflection that I could have been clearer that I meant it was crucial for someone considering their own work in the context of what you were writing, though the general argument itself would not be undermined without the classifying information.
Interesting piece. However, I’m not sure this tells us anything particularly new. Publication bias is a very well studied phenomenon, and we already know that small negative studies are less likely to be published than small positive studies. I suspect most of the “small study bias” just comes down to publication bias.
Yeah — it’s definitely something that people already think happens but this is definitely a new piece of evidence. It’s the first study to show the details of this bias with anything other than binary outcomes. And the funnel plots are really quite a striking way to illustrate the phenomenon too.
Here’s what the authors write about their study and the state of the literature: