In December 2018, the medical journal The BMJ published a study with a startling result. A team led by Robert Yeh ran a randomized trial of parachutes. People jumped from aircraft with either a parachute or an empty backpack. The parachute did not lower the rate of death or major injury at all [1].

The paper was satire. It ran in the journal’s Christmas section, which is known for playful papers. The authors wrote that their trial “satirically highlights some of the limitations of randomized controlled trials”. Yet the joke was built with care, and it makes a real point.

My view is simple. A study can follow every rule and still answer a question that does not matter for your life. So when a headline says a treatment worked or failed, ask one question first. Who was studied, and under what conditions?

How did the parachute trial work?

The team screened 92 aircraft passengers who were 18 or older. They asked each one whether they would be willing to jump at the aircraft’s current height and speed. Only 23 people agreed, and the researchers assigned each of them by chance to one of two groups.

That is what makes it a randomized trial: chance, like a coin flip, decides who gets the treatment. It is the strongest way we have to test whether a treatment causes a result.

One group jumped wearing a parachute. The other group jumped wearing an empty backpack. The researchers then checked for death or major injury on hitting the ground. They used a standard trauma scale that doctors use to rate how badly someone is hurt.

Why did nobody get hurt?

The result was zero percent in both groups. Nobody with a parachute was badly hurt, and nobody with a backpack was badly hurt either. On paper, the parachute made no difference.

Here is the catch. The people who agreed to jump were sitting in a biplane or a helicopter. Their aircraft were parked on the ground. The average height was 0.6 meters, about 2 feet, and the speed was zero.

Now compare the 69 people who were screened but did not take part. Every one of them was on a jetliner. On average, their planes were 9,146 meters up, roughly 30,000 feet, and moving at 800 kilometers per hour.

The authors state it directly. The trial “was only able to enroll participants on small stationary aircraft on the ground”. They also note that most of the people who jumped were members of the study team.

That is why the finding is true and useless at the same time. The trial answered its own narrow question correctly. It just was not the question anyone cares about.

A good study can still ask the wrong question

Scientists have a word for this problem. They call it generalizability. It means how well a study’s results carry over to other people and other settings.

A study can be rigorous and still have poor generalizability. The coin flip in a randomized trial makes the two groups fair to each other. It does nothing to make the people in the study look like the people outside it.

In the parachute trial, the two groups matched each other well. The jumpers were also similar to the people who did not take part in age, sex and medical history. The biggest difference was the one that mattered most: how high and how fast they were.

The authors point to a real pattern behind the joke. When doctors already believe a treatment works, trials may end up enrolling people who seem less likely to benefit [1]. Doctors may keep patients out of a trial because it feels wrong to deny them the treatment. Those who seem to need it most are the hardest to include. The trial can then find no effect, even if the treatment helps the people who need it most.

They also note that earlier research has suggested trial patients are at lower risk than patients in everyday care. That gap may make a treatment look weaker in a trial than it would be for sicker people.

Where did the joke start?

The 2018 trial answered an older joke. In 2003, the same journal published a mock review by Gordon Smith and Jill Pell [2]. A review like this searches the research on a question and gathers every trial that exists. They searched major medical databases for randomized trials of parachutes and found none.

Their target was critics who object to using treatments backed only by observation instead of trials. With a straight face, they suggested that the strongest believers in trials should organize and take part in a placebo-controlled trial of the parachute.

Yeh’s team wrote that their study was partly a response to that call. Their answer was clever. Yes, you can randomize people to jump without a parachute. You just have to pick jumps where it does not matter.

What the joke does not mean

It would be easy to take the wrong lesson from this. The parachute trial does not say that randomized trials are useless. The authors wrote that such trials “remain the gold standard for the evaluation of most new treatments”.

Their real warning is about reading. They argue that making sense of a trial takes more than a quick look at the summary. The headline result of this paper is that parachutes did not reduce death or injury. You have to read past that line to learn the aircraft were parked.

They even offered a fix. Studies should more often compare the people who took part with the people who did not. They should also report that comparison consistently. That is exactly how the parachute trial exposed its own catch.

The team also tried to register the trial with an official trial registry. The registry declined, saying the research question lacked scientific validity. The authors wrote that they agreed with that decision.

This matters for everyday health news. A drug trial may leave out older adults or people with several illnesses. A diet study may last only a few weeks. A test may be checked in a hospital, far from a regular clinic. None of that makes the study wrong. It just limits who the answer applies to.

How to read the next health headline

When a headline says a treatment did not work, look for three details. First, who took part: their age, health and how sick they were. Second, what the setting was: a hospital, a lab or everyday life. Third, how the study’s conditions compare with yours.

The same goes for a headline that says a treatment worked. A good result in young, healthy volunteers may not hold for someone older or sicker.

The parachute trial’s closing advice sums it up. The authors “confidently recommend that individuals jumping from small stationary aircraft on the ground do not require parachutes”. For anyone higher up, they asked readers to use judgment. That is good advice for any study: know the height of the plane before you jump to a conclusion.

References

[1] R. W. Yeh, L. R. Valsdottir, M. W. Yeh, C. Shen, D. B. Kramer, J. B. Strom, et al., “Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial,” BMJ, vol. 363, Art. no. k5094, Dec. 2018, doi: 10.1136/bmj.k5094.

[2] G. C. S. Smith and J. P. Pell, “Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials,” BMJ, vol. 327, no. 7429, pp. 1459-1461, Dec. 2003, doi: 10.1136/bmj.327.7429.1459.