A fair test, in the wild

A new JAMA trial tested a single nasal spray of a short-acting psychedelic for depression that hasn't responded to standard treatment. Here's how to read it with the tools from my last post: what it found, how big the effect is, how it compares with ketamine, esketamine, psilocybin and standard antidepressants, and what usually happens between phase 2 and phase 3.

By Reid Robison, MD. 4,061 words, with the results side by side, the effect over eight weeks, who could tell what they got, a comparison chart, the phase 2 to phase 3 track record and a phase 3 power calculator.

In my last post, A fair test, I walked through how a double-blind trial works, why placebo response is so strong in psychiatry, and the handful of numbers that let you judge a study. This week a good test case landed: a phase 2b trial of mebufotenin, better known as 5-MeO-DMT, given once as a nasal spray (the drug is called BPL-003), published in JAMA on October 26.¹ So let's use the tools on it, the way you'd read any new study.

A note on where I stand, because it matters here. I was the principal investigator at one of the trial's 38 sites, Cedar Clinical Research in Draper, Utah, and I've been a paid consultant to Beckley Psytech, the company that ran it. Everything below comes from the published paper, its online supplements and the company's public announcements, and I'll try to be as hard on it as I would be on anyone else's.

1The five questions

At the end of the last post I offered five questions to ask of any study. Here they are, answered for this one.

1. Compared with what? A 0.3 mg dose, low enough to be "subperceptual," meaning people shouldn't feel a psychedelic effect. It's a placebo-like control that looks and is given identically.¹ Verdict: a real comparison group. Good.
2. Who decided who got what? Randomization, stratified by region. It was planned as equal thirds, then changed partway through to favor 12 mg and 0.3 mg, which is why the 8 mg group is smaller; a sensitivity analysis found similar results under both schemes.¹ Verdict: chance decided. Good.
3. Who knew? The trial was double-blind, and trained central raters, who weren't in the dosing room, scored the MADRS remotely by video. But most participants could tell, and the raters often guessed right too (section 4).¹ Verdict: well designed, imperfectly blind. The honest weak point, as with every psychedelic trial.
4. How big, and for how many? A 5 to 6 point MADRS advantage; response (symptoms cut by at least half) in 34 to 35 percent against 12 percent; remission (a MADRS score of 10 or less) in 18 to 26 percent against 8 percent.¹ Verdict: a moderate effect for a meaningful minority.
5. Has anyone else found it? Partly. A separate trial of an inhaled form of the same molecule, by a different company, was also positive.⁴ But this specific product hasn't been tested in phase 3 yet. Verdict: promising, not yet replicated.

2What it found

The participants were adults aged 18 to 75 with moderate to severe major depression that hadn't responded to two to five antidepressant trials in the current episode. Their average age was 41, and 59 percent were women. Anyone on an antidepressant came off it under supervision at least two weeks before dosing. Everyone, in every arm, got the same manualized psychological support: three preparation visits, a facilitator in the room during dosing, three follow-up visits in the next two weeks and then visits every two weeks.¹ The pre-specified primary comparison was 12 mg against 0.3 mg on the MADRS, a 60-point clinician-rated depression scale, at day 29.

Applying the numbers from last time: the 12 mg result, a 5.3-point advantage with a 95 percent confidence interval of 1.8 to 8.9 points and p = .003, means the effect is very unlikely to be chance, but its true size could be anywhere from small to large.¹ The confidence interval matters as much as the p-value.

Converting to the units from last week (these are my calculations from the reported means and standard deviations; the paper doesn't report effect sizes): the effect size is roughly 0.49 for 12 mg and 0.56 for 8 mg. That's moderate, and in the same neighborhood as the median psychiatric drug (0.41) and the median drug across general medicine (0.37) in one well-known comparison.⁵ For response, the number needed to treat is about 4 for 8 mg (35 versus 12 percent) and about 5 for 12 mg (34 versus 12). For remission it's about 6 for 8 mg and 10 for 12 mg.

Notice that 8 mg did at least as well as 12 mg, with fewer side effects. The company chose 8 mg for phase 3.⁶ The paper's other measures pointed the same way: clinician-rated overall severity and quality of life improved more on both active doses, and a self-rated depression scale did on 12 mg.¹ The authors attribute the lower remission rate on 12 mg to a tolerability trade-off: nausea, anxiety and panic attacks were more common at the higher dose. More drug isn't always better, and dose-ranging is one of the main jobs of phase 2.¹

3Over eight weeks

One dose, then eight weeks of blinded follow-up. The separation appeared by day 2 and lasted through day 57, the last blinded visit. The adjusted advantage for 8 mg was 4.7 points on day 2, 6.3 on day 8, 6.2 on day 29 and 5.5 on day 57; for 12 mg, 5.0, 6.2, 5.3 and 5.0. At day 57, 28 percent on 8 mg and 32 percent on 12 mg were still responders, against 11 percent on the control.¹ The authors point out that the advantage kept growing for the first few weeks rather than peaking in the first day or two, and showed no clear fade by week 8.¹

Durability is one of the most important features of a single-dose treatment, and one of the hardest to judge. Eight weeks is encouraging; it isn't maintenance. In an open-label extension, everyone who continued got a second 12 mg dose at week 8, and the company reported further improvement, but without a control group those numbers can't tell us how much was the drug.⁷ How often people will need retreatment, and whether second and third doses keep working, are the next questions.

4Who could tell?

This trial did something I asked for in the last post: it reported how well the blind held. Participants, clinicians and the MADRS raters were each asked which treatment they thought the participant got.¹

Participants could tell: 90 percent on the active doses guessed active, and 70 percent on the control guessed control. That was expected; a dose of 5-MeO-DMT is unmistakable. The more important row is the raters. The authors write that blinding was maintained for the central raters, and it clearly held much better for them than for participants or clinicians.¹ My read of the same figure is a little more cautious: the raters never saw a session, yet they guessed right about 60 percent of the time, against 50 percent by chance, and one in eight said they didn't know. Partly blinded is how I'd put it. Raters can pick up clues from how people describe their week, which is very hard to prevent.

The company's own plan addresses this. Its phase 3 trials are set to add a mid-range dose that people can feel but that's less intense, so "I felt something" no longer reveals the assignment, the same approach the lysergide phase 3 trials took with their 50 microgram arm.¹⁸

Two findings cut the other way. Participants' expectations before dosing, measured with a standard scale, had essentially no relationship with how much they improved on the active doses (correlations of 0.04 and −0.12). And the intensity of the experience, measured with a mystical-experience questionnaire, was much higher on the active doses, but the authors found no correlation between that score and improvement at day 29. They suggest either that something other than the subjective experience drives the benefit, or that the questionnaire doesn't capture this drug's experience well.¹ Neither finding proves the effect isn't partly expectancy, but they're the kind of checks you want to see.

One small, human detail from the safety tables makes the point better than any statistic: one of the two serious adverse events in the trial was a person on the 0.3 mg control who was hospitalized for worsening depression, attributed to disappointment on the dosing day.¹ Knowing you didn't get the drug has effects of its own.

5Subtract the control

The control arm improved too: 5.5 points on the MADRS, and 12 percent responded.¹ That's everything from the "Why people get better on nothing" section of the last post at work: regression to the mean, hope, attention, the structure of a study. It's actually on the low side for depression trials, which fits treatment-resistant depression, and a control dose that most people could tell was inactive.

The same subtraction applies to side effects. At least one adverse event was reported by 76 percent on 8 mg, 85 percent on 12 mg and 73 percent on 0.3 mg.¹ So the share plausibly caused by the drug is roughly 3 points at 8 mg and 12 points at 12 mg, not 76 or 85. Individual side effects show the subtraction even more clearly. Headache was just as common on the control (28 percent) as on the drug (26 and 33 percent), so it's mostly not the drug. Nausea was not: 33 percent on 8 mg and 40 percent on 12 mg, against 3 percent on control. Vomiting, nasal pain or discomfort and raised blood pressure were also clearly higher on the active doses. Of all adverse events, 99.7 percent were mild or moderate, and nobody stopped the study because of side effects.¹

Blood pressure and heart rate rose briefly. Average systolic pressure peaked at about 146 to 147 on the active doses, against 134 on control, within the first 10 minutes, and was back near baseline by about an hour; average heart rate peaked at 93 and 102 beats per minute.¹ That's a reason for careful screening of people with heart disease or uncontrolled hypertension. On suicide-risk monitoring, nobody in any arm had suicidal behavior, and the few reports of suicidal thoughts with intent came only from the control arm.¹ Both serious adverse events (one in the 8 mg arm, one on control) were judged unrelated to the drug.¹ In the open-label extension, one drug-related serious adverse event was reported and resolved with inpatient monitoring.⁷

One more practical number, with an asterisk. Readiness for discharge was checked every 30 minutes from 90 minutes after dosing, and 85 percent of people on 8 mg and 78 percent on 12 mg met the criteria by two hours; the average was just under two hours. But the protocol didn't let anyone leave before four hours.¹ Psilocybin dosing days in trials have run six to eight hours. If a real-world label lands closer to two hours than four, it matters for clinics, staffing and cost, which are some of the biggest practical barriers to these treatments.

6Compared with what we already have

Here's where reading carefully matters most. It's tempting to line up the headline numbers from different trials and rank them. The chart below does line them up, because seeing them together is useful, but every row was measured at a different time, against a different comparator, in a somewhat different population. Tap any row to see what it was compared with.

A few honest readings of that chart:

• Versus psilocybin. This trial's 5 to 6 points at four weeks is in the same range as psilocybin's 6.6-point phase 2b result at three weeks, against a similar low-dose control.² Psilocybin's two phase 3 trials then came in at 3.6 and 3.8 points at six weeks.⁹ That pattern is the subject of the next section.
• Versus esketamine (Spravato). In its positive phase 3 trial, esketamine beat placebo by 4.0 points at four weeks, but with repeated twice-weekly dosing and with both groups also starting a new oral antidepressant, which gives the comparison group a real treatment too.¹⁰ A numerically bigger bar here doesn't mean a better drug.
• Versus IV ketamine. Ketamine's large advantage over midazolam was measured at 24 hours.¹¹ That answers a different question, rapid relief, from this trial's four-week result.
• Versus the inhaled version of the same molecule. GH Research's GH001 showed a very large day-8 difference against an inert placebo, with up to three individualized doses in one day.⁴ An inert placebo is far easier to see through than a 0.3 mg dose, and day 8 is early, so the bars aren't comparable.
• Versus standard antidepressants. They're measured mostly in broader, less treatment-resistant groups, usually on the Hamilton scale, with daily dosing. In the FDA's pooled data, the average advantage was 1.75 Hamilton points, with about 15 percent of people getting a substantial benefit beyond placebo.¹² Different scale, different population, so no bar.

What I'm comfortable saying: in people with treatment-resistant depression, one dose produced an effect in the same range as the best-studied rapid-acting treatments, with an acute phase that may allow a much shorter treatment day than psilocybin's. What I'm not comfortable saying: that it's better than any of them. That claim would need head-to-head data, which nobody has.

7From phase 2 to phase 3

Phase 2 asks whether a drug seems to work and at what dose. Phase 3 asks whether it really does, in larger, broader trials that regulators will judge. Psychiatry has the hardest phase 2 in medicine: in an industry analysis of programs from 2006 to 2015, only 24 percent of psychiatric drugs in phase 2 advanced to phase 3, the lowest of 14 disease areas.¹³ And drugs that do advance often show smaller effects in phase 3.

There's a statistical reason, sometimes called the winner's curse. Phase 2 trials are small, so their results bounce around. Programs move forward when the bounce happens to look good. Phase 3 then measures the drug again, in more people, at more sites, often with higher placebo response, and the estimate tends to settle back toward its true value. Two recent psychedelic programs show how that can look.

Psilocybin's effect roughly halved, from 6.6 points in phase 2b to 3.6 and 3.8 in phase 3, though the phase 3 trials measured at six weeks instead of three, and one used an inert placebo.²⁹ Both phase 3 trials still succeeded. The lysergide (MM120) program, which I worked on, held its phase 2b effect size of 0.81 in one phase 3 trial and came in at 0.64 in the other.¹⁴¹⁵⁸ Shrinkage isn't failure. A smaller, real, replicated effect is how a treatment earns its place.

So what does phase 3 need to look like for mebufotenin? Here's a rough way to think about it. Slide the effect phase 3 might actually show, and the number of people per arm, and see the chance that a trial of that size comes out significant.

Try 6 points, the phase 2 result: a modest trial has excellent power. Then try 3 or 4 points, closer to what psilocybin showed in phase 3: you need roughly twice as many people, or more, for the same confidence. Sponsors plan for the shrink, and that's why phase 3 trials are bigger. The company has announced two parallel phase 3 trials, each with a 12-week placebo-controlled core and a 52-week open-label extension with retreatment, with results expected in early 2029. The drug has FDA Breakthrough Therapy designation, which speeds the process but doesn't lower the bar for approval.³

8What would make me more confident

The authors list the limitations themselves, which is a good sign: functional unblinding; a smaller 8 mg group after the mid-trial change to randomization, which limited power for some secondary outcomes; requiring people to come off antidepressants, which may not match future practice (though in an extra analysis the effect was similar whether or not people had washed out); limited ethnic diversity; possibly too few of the most severely treatment-resistant patients; possibly underreported past psychedelic use; and no separate test of the psychological support.¹ To those I'd add the usual phase 2 caveats from the last section.

And one more of the five-question habits: ask who paid and who wrote it. Beckley Psytech funded the trial, and its employees were involved in the design, the analysis, the writing and the decision to publish. Several authors are company employees or consultants, some are named on a pending patent for these doses, and a medical writing firm paid by the company helped draft the paper. All of it is disclosed, and it's standard for an industry phase 2 trial.¹ It's also exactly why independent replication, and phase 3 trials judged by regulators, matter.

What would move me most:

• Replication in phase 3, ideally with an effect that holds up at 12 weeks, in a broader population.
• Retreatment data with a comparison group: how often people need another dose, and whether it keeps working.
• A better-protected measurement. This trial reported its blinding, which many trials don't.¹⁶ The next step is analyses showing whether the effect holds among people who guessed wrong, and ways to keep raters from inferring the assignment.
• Longer-term safety, including blood pressure and anything rare enough to need thousands of patients to see.
• Real-world feasibility: whether a two-hour visit stays two hours outside a trial.

My read, with my biases on the table: this is a well-run, clearly positive phase 2b trial of a treatment that could make psychedelic-style care much less time-intensive, if it holds up. That "if" is what phase 3 is for. And that's the real point of reading studies this way. You don't have to take my word for it, or the company's, or a headline's. You can ask the five questions yourself.

References