A fair test

How a double-blind drug trial actually works, why so many people get better on placebo in psychiatry, how trials (including psychedelic ones) still find the truth, how to read side effects, and the handful of numbers that let you judge a study yourself.

By Reid Robison, MD. 5,612 words, with a trial you can walk through step by step, a regression-to-the-mean machine, placebo response by condition, a trial simulator you can run a hundred times, a side-effect subtractor, an effect-size explorer and a jargon decoder.

We are drowning in claims. A video tells you a common painkiller causes autism; another tells you a supplement cures depression; a third says antidepressants are no better than sugar pills. Some come from strangers, some from companies with something to sell, and lately some have come from the federal agencies we used to treat as the referee. Most people don't have time to read the studies. That makes the question underneath all of it urgent: who do you trust?

My answer has two parts. In a relationship, including the one between a patient and a therapist, trust is how healing happens, and I'd never tell you to give that up. But a claim about what a medicine or a vaccine does is a different kind of thing. There the question isn't whether the person making it is kind or sincere. It's whether they followed a procedure that would have caught them if they were wrong. Good, caring, honest people fool themselves all the time; I have too. So trust people, and check claims.

This post is about that procedure as it's used in psychiatric drug trials, which is the world I've worked in for more than 15 years. I've written before about how placebo and nocebo work in the body, why the placebo response is so strong in brain medicine, and how a single adverse event travels from a clinic note to a drug label. This is the piece that ties them together: what a trial is, how it fights placebo, how it can still work when the drug announces itself, and the few numbers you need to read the result.

1Who do you trust?

You deserve to know my lens, so I'll say it plainly. I've been dismayed by a series of federal health decisions over the past two years that, in my view, moved away from the evidence. A few examples, each documented: in May 2025 the administration's "MAHA" report on childhood chronic disease cited studies that did not exist; seven nonexistent references were quietly removed at first, and within a week at least 18 citations had been changed.⁷ In June 2025 the HHS secretary removed all 17 members of the CDC's vaccine advisory committee.⁸ In September 2025 the President and the secretary announced that acetaminophen in pregnancy is linked to autism, even as the FDA's own notice to physicians that evening said "a causal relationship has not been established."⁹ In November 2025 the CDC's vaccine-safety page was changed to say that "vaccines do not cause autism" is "not an evidence-based claim," with no new research cited.¹⁰ And in January 2026 the CDC cut the universally recommended childhood vaccines from 17 to 11 without a public review of the data; in March a federal judge stayed that change and the new advisory committee's appointments, and the government has appealed.⁸¹¹¹²

You may see some of that differently, and you're welcome here either way. What I'm asking for isn't agreement with me. It's that we judge every claim by the same rules, whether it comes from an agency, a drug company, a wellness influencer or a psychiatrist with a blog. Those rules are the part of science that isn't really debatable, because they aren't opinions about results. They're agreements about how to avoid fooling ourselves before we see the results.

Richard Feynman put the first rule in a 1974 commencement speech: you must not fool yourself, and you are the easiest person to fool.¹³ Every rule below exists because someone, usually a sincere and smart someone, fooled themselves in exactly that way.

The rules of a fair test

1. Compare. "I took it and got better" can't tell you whether the treatment did anything, because most conditions change on their own. You need a group that didn't get it.Guards against: natural recovery, regression to the mean.
2. Let chance pick who gets what. If doctors or patients choose, the sicker, healthier, richer or more hopeful people end up in one group. A coin flip makes the groups alike in everything, including things nobody thought to measure.Guards against: selection bias and confounding.
3. Hide who got what. If patients know, their expectations change how they feel. If raters know, their hopes change how they score.Guards against: expectancy and observer bias.
4. Say what you'll measure before you look. Registering the main outcome and analysis plan in advance stops anyone from picking the one result out of twenty that happened to look good.Guards against: cherry-picking and p-hacking.
5. Count everyone. People who drop out because the drug made them sick still count. Analyzing only the people who stuck with it flatters any treatment.Guards against: attrition bias.
6. Do it again. One trial can be a fluke. A finding that holds up in different trials, run by different people, is how knowledge earns trust.Guards against: chance and error.

Here's why these rules matter more than anyone's credentials. The acetaminophen question is a perfect example. Many observational studies found that children whose mothers took acetaminophen in pregnancy were slightly more likely to be diagnosed with autism or ADHD. But mothers who take more acetaminophen differ in many ways: more fevers, more infections, more pain conditions, different genes. A 2024 Swedish study of nearly 2.5 million children used a clever version of rule 2 and compared siblings, children of the same mother, one exposed and one not. Within families, the link disappeared.¹⁴ Nothing about that conclusion depends on whom you voted for. It depends on whether the comparison was fair.

The same rules are also how science corrects itself. The 1998 paper that started the vaccine-autism scare was retracted by The Lancet in 2010 after its methods and data were exposed, and many large studies since have found no link.¹⁵ That's not a sign that science can't be trusted. It's the procedure working, slowly, the way it's supposed to.

2Anatomy of a double-blind trial

Almost every psychiatric medicine you've heard of was approved on the strength of randomized, double-blind, placebo-controlled trials. "Double-blind" means neither the participant nor the people evaluating them know who got the drug. Walk through one below; each step exists to block a specific way of fooling ourselves.

A few details trip people up. The placebo is made to be indistinguishable from the drug: same size, color, taste and schedule. The randomization list is generated by a statistician and locked away; the site can't see it, and in most trials neither can the sponsor's study team. The people who score the symptoms are often different from the ones who see the participant for safety, and increasingly they're remote, independent "central" raters who never meet the participant in person. Before anyone unblinds the data, the primary outcome and the analysis are already locked in a statistical analysis plan, and the trial is registered publicly at ClinicalTrials.gov.

3Why people get better on nothing

The phrase "placebo response" is misleading, because most of what happens in a placebo arm isn't caused by the placebo. It's everything that makes people improve over a few weeks in a study, whether or not they take anything. There are at least five ingredients:

• Natural course. Depression and anxiety wax and wane. People often seek help, and enroll in studies, at their worst.
• Regression to the mean. If you enroll people because they scored high today, many will score lower next time just by chance. More on this in a second.
• Expectation. Believing you might be getting an effective treatment changes how you feel, through real brain mechanisms (I've written about those here).
• Care and attention. Regular visits with kind, attentive staff who ask how you're doing are therapeutic in themselves. That's a feature of good care, and a headache for a trial.
• Measurement drift. Participants learn what the questions are getting at; raters who want the trial to work may score a little hopefully; and if a minimum score is needed to enroll, scores at screening can drift up.

Regression to the mean is the least intuitive, and one of the most important, so here it is as a machine. Imagine a thousand people whose true depression level stays exactly the same for the whole study. Every rating includes some random wobble (a bad night's sleep, a rough day, a different rater). The trial only enrolls people who score above a cutoff. Then everyone is rated again a few weeks later, with no treatment at all.¹⁶

That apparent improvement, with no treatment at all and nobody truly changing, is why every trial needs a comparison group, and why a stricter cutoff or a noisier rating scale can make a placebo arm look better.

4How strong is placebo, condition by condition

In depression, the best long-run picture comes from a review of 252 placebo-controlled antidepressant trials with more than 26,000 people on placebo: since about 1991, the placebo response rate (meaning at least a 50 percent drop in symptoms) has stayed between 35 and 40 percent.¹ It had been climbing before that. An earlier review of 75 trials published from 1981 to 2000 found placebo response averaged 29.7 percent, ranging from 12.5 to 51.8 percent, against 50.1 percent on drug, and both rose with publication year.¹⁷ That range is the first lesson: the same condition can produce a placebo response of 13 percent in one trial and 52 percent in another.

In acute schizophrenia, a review of 167 trials and more than 28,000 people found at least minimal improvement in 51 percent on antipsychotics and 30 percent on placebo, and "good" response in 23 percent against 14 percent. Placebo response rose over the decades while drug response held steady, and that rising placebo response was one of the few factors that explained shrinking effect sizes.¹⁸

To compare conditions fairly, you need the same yardstick. In 2024 a team in Germany pulled ten trials for each of nine psychiatric conditions, almost 10,000 people on placebo in all, and measured how much symptoms improved from start to finish on placebo, as a standardized effect size.²

The pattern makes clinical sense. Depression and generalized anxiety are conditions where hope, support, sleep, routine and the passage of time all matter a great deal, and where the main outcome is how a person feels and reports. Schizophrenia and OCD have more fixed, self-perpetuating cores; placebo still helps, but the illness pushes back. The authors put it carefully: placebo factors can improve symptoms, but their reach is limited by the underlying illness.²

5The stats problem, made visible

Here's the bind. If half of people improve on the drug and 35 to 40 percent improve on placebo, the drug's measurable advantage is 10 to 15 percentage points. Random chance can easily make a gap that size appear, or disappear, in a trial of a hundred people. This is why good drugs regularly "fail" trials, and why one positive trial is not proof.

Run trials yourself. Set the true response rates and the number of people in each arm, then run one trial, or a hundred at once. The second button is the one that teaches.

Try three experiments. With "typical trial," run 100 trials and notice how many come out "not significant" even though the drug truly works; that's the power problem, and it's why sponsors run more than one trial and the FDA usually asks for at least two positive ones. With "drug does nothing," run 100 and notice that about 5 of them come out "significant" anyway; that's what p < 0.05 means. And raise the placebo response while keeping the drug's advantage the same: the trials get harder to win, because the same gap sits on top of more noise.

Some trial features push placebo response up. In the 252-trial review, each extra week of trial length raised placebo response slightly (relative risk 1.03 per week), and multicenter trials had 32 percent higher placebo response than single-center trials.¹ More sites mean more raters, more variability and more pressure to enroll.

6How trials fight back

Nobody can, or should, remove the human care that drives part of the placebo response. What trials can do is keep the noise down so the drug's signal is easier to see. These are the main tools, roughly in order of how often I see them in protocols today.

Blinded, independent raters

A trained rater who doesn't know the treatment, the visit number or how the participant is doing on safety scores the main outcome, often by phone or video from outside the site. It cuts rater hope and drift.

Rater training and structured interviews

Raters are certified on the scale, re-calibrated over time, and use structured interviews with fixed anchors so that "moderate" means the same thing in Utah and Ohio.

Separate scales for entry and outcome

Using one scale to decide who qualifies and a different one as the main outcome reduces the screening-score inflation that feeds regression to the mean.

Smarter designs

The sequential parallel comparison design, proposed by Maurizio Fava and colleagues, runs a placebo-controlled phase, then re-randomizes the people who didn't respond to placebo, a group whose placebo response is much lower, and combines both phases.¹⁹

Fewer, better sites

Because multicenter trials carry higher placebo response, some sponsors now favor fewer experienced sites over many new ones.¹

Setting expectations

Consent conversations that are honest about the chance of getting placebo and about what a study visit is (a measurement, not therapy) help keep expectations realistic.

Counting everyone

Analyzing every randomized person (intention to treat), with modern methods for missing data, keeps dropouts from flattering either arm.

None of these is magic. Placebo response still varies a lot from trial to trial, which is why the same drug can win one study and lose the next, and why I trust a pattern across several trials far more than any single one.

7Psychedelics and the blinding problem

Blinding is never perfect. Even in antidepressant trials, where the drugs are subtle, people sometimes guess right. A review of antidepressant trials from 2000 to 2020 found that fewer than 10 percent even reported checking. In the nine trials with usable data, agreement between guesses and actual assignment was modest (a kappa of 0.21 for patients and 0.17 for raters, where 0 is pure guessing and 1 is perfect), with little evidence that guessing right inflated results.²⁰

Psychedelics are a different order of problem. If you've taken 100 micrograms of LSD, you know. That makes the classic double-blind impossible for the participant, and it raises a fair question: is the benefit the drug, or the excitement of knowing you got it?²¹

I can describe how this played out in the trial I know best, the phase 2b study of MM120 (lysergide) in generalized anxiety disorder, which I worked on and was first author of the JAMA paper. About 85 percent of participants who got the drug correctly guessed they had, and more than 65 percent of those on placebo guessed placebo. But the people who scored anxiety were independent central raters who interviewed participants by phone without knowing the dose or the visit, and they stayed blinded in more than 80 percent of their assessments.⁴

What is controlled, and what isn't

How the results can still be valid

No single trick solves it. The case rests on several lines of evidence pointing the same way:

• Dose arms. In the MM120 phase 2b, people on 25 and 50 micrograms could often tell they'd gotten something, yet those doses did not separate from placebo; 100 and 200 micrograms did.⁴ If knowing alone drove the effect, the lower doses should have worked too. Phase 3 trials added a 50 microgram arm partly so that "I felt something" wouldn't reveal the dose.²²
• Blinded measurement. Central raters who never see the session score the outcome.
• Active placebos. Some trials use a drug that causes noticeable effects (niacin, a low dose, or a different active compound) so that "I feel something" is less informative.²¹
• Durability. Expectancy tends to fade. Benefits that persist for months after a single dose, as in the MM120 trial's week-12 results, are harder to explain by excitement alone.⁴
• Measuring expectancy. Asking participants their expectations before dosing, and their guesses afterward, lets analysts check whether the effect holds among people who guessed wrong or had low expectations.
• Replication. Results that repeat across independent phase 3 trials, in different places and populations, are the strongest answer.²³²²

The FDA's guidance for psychedelic trials, finalized in July 2026, asks sponsors to address exactly these issues, including how they'll limit and measure functional unblinding.⁵ My own view: the blinding problem is real, it's been taken seriously, and the right response is not to wave away the results but to read them with this weakness in mind. That's what I'd ask of any study.

8Side effects: the placebo column

If placebo can make people feel better, it can make them feel worse, and the numbers are bigger than most people expect. In an individual-participant analysis of 2,457 people on placebo in duloxetine depression trials, 63.9 percent reported at least one adverse event, and 4.7 percent dropped out because of them.³ Across 58 antipsychotic trials, 66.3 percent of people on placebo reported an adverse event, most often insomnia and headache, and 7.2 percent withdrew because of adverse events.²⁴

Placebo side effects even take on the shape of the drug people expect. In older trials of tricyclic antidepressants, the placebo groups reported more dry mouth, drowsiness and constipation, the tricyclics' famous side effects, than placebo groups in SSRI trials did.²⁵

This is why every side-effect table on a drug label has two columns. The number that tells you what the drug causes is the difference. Try it.

Notice how much the subtraction changes the story. Dry mouth on escitalopram looks like a 6 percent problem, but 5 percent of people on placebo report it too. Ejaculation problems look like a 9 percent problem, and almost nobody on placebo reports them, so nearly all of it is the drug. Stopping because of side effects happened in 6 percent of people on escitalopram and 2 percent on placebo.²⁶

Which conditions have more placebo side effects?

The same German group looked at dropouts from placebo arms across their nine conditions. Overall dropout was highest in schizophrenia, where 41 percent of people on placebo left early, with lack of improvement a leading reason, and lowest in PTSD, depression and ADHD, around 17 percent. Dropout specifically because of adverse events was highest in OCD, at about 7 percent, with panic disorder also high, and lowest in depression and ADHD.²⁷

Why those differences? We don't fully know, and I want to be careful here. Some plausible reasons: in schizophrenia and the anxiety disorders, the illness itself produces insomnia, restlessness and physical discomfort that get recorded as adverse events. People with panic disorder are often acutely attuned to body sensations, which is part of the condition. And in one large depression analysis, placebo side effects weren't tied to what the consent form listed, or to people's past medicines; features of each study predicted them best.³ The practical lesson is the same everywhere: before blaming a new medicine, ask how common that symptom was before it started, and how common it was on placebo.

9The numbers that matter

You can read most results with five ideas.

p-value

If the drug truly did nothing, how often would chance alone produce a difference at least this big? "p < 0.05" means less than 5 percent of the time. It does not tell you how big or important the effect is, or the probability that the drug works. The American Statistical Association felt the need to say so in an official statement.⁶

Confidence interval

The range of effects consistent with the data. A drug that beats placebo by 3 points, with an interval from 0.5 to 5.5, has a real but uncertain effect. Narrow intervals come from big, clean trials.

Effect size

How big the difference is, in standard units so you can compare across scales. Roughly: 0.2 small, 0.5 medium, 0.8 large. The explorer below makes it concrete.

Response and remission

Response usually means at least a 50 percent drop in symptoms; remission means symptoms low enough to count as well. Response rates are intuitive but depend on where the line is drawn.

NNT and NNH

Number needed to treat: how many people need the drug, instead of placebo, for one more to respond. Number needed to harm is the same idea for a side effect. In the MM120 phase 2b, response at week 12 was 65.0 percent on 100 µg versus 30.8 percent on placebo, an NNT of about 3.⁴

Two things surprise people. First, psychiatric drugs are not unusually weak. A review of meta-analyses comparing 16 psychiatric drugs with 48 drugs used across general medicine found effect sizes in the same range: a median of 0.41 for psychiatric drugs and 0.37 for the general medicine drugs.²⁹ Some psychiatric treatments are among the larger effects in all of medicine: in children with ADHD, amphetamines beat placebo with an effect size of 1.02 on clinician ratings.³⁰ Second, averages hide people. In the FDA's own data from 232 antidepressant trials and 73,388 people, the average advantage over placebo was small, 1.75 points on the Hamilton depression scale. But a large improvement occurred in 24.5 percent of people on drug and 9.6 percent on placebo; the authors estimated that about 15 percent of people get a substantial benefit beyond placebo.³¹ That's the honest picture: modest on average, meaningful for a real minority, and the job of good care is to find out quickly whether you're in that minority. In the largest network analysis of antidepressants, every one of 21 drugs beat placebo, with odds ratios for response from 1.37 to 2.13.³²

10A jargon decoder

Abstracts on PubMed are written in a compressed dialect. Type a word you've seen, or browse.

11Five questions before you believe a study

You won't always have time to read the paper. These five questions get you most of the way, and they work on a headline, a press release or an abstract.

1. Compared with what? No comparison group means you can't separate the treatment from time, hope and regression to the mean.
2. Who decided who got what? Look for "randomized." If people chose their own treatment, or the study is observational, be cautious about cause and effect.
3. Who knew? Look for "double-blind," and for blinded raters. If everyone knew, expectations are baked in.
4. How big, and for how many? Ask for the effect size, the response rates in both arms, or the NNT, not just "significant."
5. Has anyone else found it? One study is a lead. A replicated finding is knowledge.

In future posts I want to take the same approach to two neighbors of this one: how to test a test (what sensitivity, specificity and a "positive" result really mean), and how to read a medical abstract without a decoder ring. For now, if you take one thing away, let it be the coin flip. Almost everything we know about which treatments work, in psychiatry and everywhere else, rests on that humble act of letting chance decide, and then being honest about what we see.

References