Significance without size: how supplement trials report their results
We read the abstracts of 2,481 landmark supplement trials. Almost all of them tell you whether an effect was "statistically significant." Far fewer tell you the one thing that matters — how big the effect was.
A supplement trial is roughly 4 times more likely to tell you an effect was significant than to tell you how large it was.
By Baher Al Hakim, Founder, SupStack · Published July 11, 2026
63.8%
report a p-value (was it significant?)
16.2%
report a confidence interval (the size)
10.5%
name an effect size
~4×
more likely to report significance than size
"Significant" doesn't mean "big"
A p-value answers a yes/no question: is this effect probably real, or could it just be noise? That is useful — but it says nothing about whether the effect is large enough to matter to you.
An effect can be statistically significant and still be tiny. With enough participants you can prove — beyond reasonable doubt — that a supplement lowers something by an amount too small to ever notice. The p-value lights up green. The real-world benefit is close to zero. Only the effect size and its confidence interval can tell those two situations apart.
p-value
"It will rain today." Probably true. Tells you nothing about whether to cancel the picnic.
Effect size + interval
"About 0.2 inches, give or take." Now you know it's a drizzle, not a downpour — and how confident to be.
When a study reports only the first kind of statement, you're told the finding is real but left to guess whether it's worth anything.
What a supplement trial's abstract actually reports
16 in 100
report a confidence interval — the effect's actual size. The rest tell you only whether it was significant.
Share of 2,481 randomized controlled trials whose abstract includes each element. Teal tells you the size of the effect; amber tells you only whether it was significant.
Tells you the size of the effectTells you only if it was significant
A p-valueWas the result significant?63.8%
A confidence intervalHow big is the effect, and how precise?16.2%
A named effect sizee.g. risk ratio, mean difference10.5%
That's a ~4× gap between significance and size.
Source: SupStack analysis of 2,481 RCT abstracts, classified by PubMed's own publication type. Detectors validated against a blind second-pass coding of 100 abstracts (F1 > 0.95).
It's a choice, not a limitation
Nothing about running a trial stops the authors from reporting an effect size. We know this because a different kind of study — a meta-analysis, which pools many trials together — does it as a matter of routine.
75%
of the 1,680 meta-analyses in the same corpus report a confidence interval — against just 16.2% of individual trials. Same field, same evidence, a completely different reporting habit. The size information exists. Most trials simply don't put it where you'll read it.
And it's barely improving
Share of RCTs reporting a confidence interval over time. It has risen, but remains the exception — the gap is real and current, not a relic of older papers.
What this means when you read a claim
1
"Statistically significant" is the floor, not the finish line
It means the effect is probably real. It does not mean the effect is big enough to change how you feel.
2
Look for the number and its range
A trustworthy result names the effect size and a confidence interval — e.g. "lowered it by 3 points (95% CI 1 to 5)." An interval that hugs zero is a weak effect, however small the p-value.
3
A lone p-value is a half-told story
If a headline or label cites only "p < 0.05" and no magnitude, the size was left out — and that's the part that decides whether it's worth your money.
Common questions
Do supplement studies prove supplements work?+–
Not on their own. A p-value (which 63.8% of trials report) shows an effect is probably real — not that it is big enough to matter. Only 16.2% report the effect size that decides that.
What is the difference between a p-value and a confidence interval?+–
A p-value tells you whether an effect is probably real (yes/no). A confidence interval tells you how big the effect is and how precise — the range the true value plausibly falls in.
Why does it matter whether a trial reports the effect size?+–
Because an effect can be statistically significant and still be too small to notice. Without the size and its interval, you can't tell whether a real result is also a meaningful one.
Does this mean the supplements don't work?+–
No. This analysis is about how trials report, not whether a supplement works. No efficacy claim is made.
Embed these charts
Free to reuse with attribution — copy the code to drop a chart (with a link back) into your article or post.
Only 1 in 6 trials reports the effect size
Significance vs. size reporting
Effect-size reporting over time
How to read this (and its limits)
- ·This is not a claim about whether any supplement works — it is about reporting, not efficacy.
- ·It measures the abstract, not the full paper. Some trials report the interval only in the body text.
- ·These are hand-picked, highly cited trials, so 16.2% is an upper bound: across all supplement trials, the real rate is likely lower.
Methodology & data+
Every abstract of 2,481 randomized controlled trials from SupStack's curated citation list was fetched from PubMed and scanned by deterministic text detectors for p-values, confidence intervals, and effect-size measures. Study types come from PubMed's own labels, not ours. The detectors were validated against a blind second pass over 100 abstracts (F1 > 0.95 on all three measures). Snapshot July 11, 2026.
Use the data
The per-study derived flags (PMIDs + detector booleans) are open and reproducible by a committed script. The underlying abstract text is PubMed's and is not redistributed.
Cite this: "Significance without size: how supplement trials report their results (2026), SupStack — supstack.me."