In December 2023, the journal Nature Communications posted a call for papers. It was for a collection called Clinical trials 2024, open for submissions until the last day of 2024. The journal asked for reports of trials that test a treatment or other intervention in people, from phase I to phase IV. A primary endpoint is the main question a trial is built to answer. The journal spelled out that its call included trials “yielding negative results for which primary end-points were not met” [1].

That is an unusual thing for a journal to put in writing. Many researchers learn early that a failed experiment is hard to publish. I think the call deserves attention for that reason alone.

A good negative result can save other people years of work and a lot of money. In medicine, it can also save the time and goodwill of patients. But I also think we should be honest about which negative results earn that praise.

In my view, a careful negative result from a study that was large enough to answer its question is a real finding worth publishing. A small study that could not give a clear answer either way should be treated differently.

What did the journal actually ask for?

The page says the journal is especially interested in papers that first report a trial’s main and secondary outcomes.

The page also makes clear that the standards stay the same. Peer review means other experts check a paper before it is published. The collection page says: “All submissions will be subject to the same peer review process and editorial processes as regular Nature Communications articles.”

The page is fair about the limits, too. It notes that the journal “cannot guarantee that any individual paper will be included in the desired collection.”

In my view, this call is a good sign, but it does not fix the problem. One call for papers cannot undo decades of habit. It also tells us nothing about how many such trials were sent in or published.

Still, researchers decide what to write up partly based on what they think editors want. Here, a journal said in writing that it wants negative trials too.

Where do the null results go?

A null result is one that finds no effect. The evidence that these results get lost is not new.

In 2012, Daniele Fanelli published a paper with the blunt title Negative results are disappearing from most disciplines and countries [2]. He searched more than 10,800 journals for papers that said they had tested a hypothesis, meaning an idea set up to be checked. He then drew a random sample of 4,656 papers from 20 fields, published between 1990 and 2007. Top general journals such as Science and Nature were left out.

For each paper, he read the abstract, and the full text when needed. He noted whether the authors concluded that their idea was supported, partly supported, or not supported. Full or partial support counted as a positive result. If a paper tested several ideas, he counted only the first one.

In 1990 and 1991, about 70 in 100 of these papers reported support. By 2007, about 86 in 100 did. The highest year was 2005, at about 89 in 100. So the share grew by more than one-fifth of where it started.

The rise was not the same everywhere. Positive results were most common, and grew fastest, in the social sciences, such as psychology and economics. Space science had the fewest positive results and was one of the few fields with a slight decline. Papers whose lead contact author was based in Asia, especially Japan, reported more positive results than papers from the United States. American papers, in turn, reported more than European ones, especially from the UK.

One detail stood out to me. Some of the steepest rises were in clinical medicine, pharmacology and molecular biology. These fields had already tried fixes such as registering trials in advance. Fanelli wrote that such efforts “have not met their objectives so far”.

Fanelli was careful about what the trend could mean. He wrote that the patterns fit two possible explanations, and that either or both could be true. One is “that research is becoming less pioneering”, for example because scientists pick questions likely to give a publishable yes. The other is “that the objectivity with which results are produced and published is decreasing”. That could mean negative results are submitted less, accepted less, or reworded into positive ones. If the second reason is right, published research looks more certain than the work behind it.

He also weighed a kinder explanation: maybe studies simply got bigger and better at finding real effects. He judged that unlikely, because other research gave no sign that studies had become better at detecting effects over those years.

His study had limits that he named himself. Fanelli sorted every paper himself, although he did not know a paper’s exact year or country while coding it. All the papers also came from a single journal database. He also warned that “a system that disfavours negative results” might “discourage high-risk projects”.

A 2014 study in Science shows more clearly where results get lost. Annie Franco, Neil Malhotra and Gabor Simonovits tracked 221 social science experiments [3]. All were survey experiments run through a program called Time-sharing Experiments in the Social Sciences, using national samples of American adults. The program kept a record of every study it ran, so the team could see what was published and what was not.

Every study had passed expert review before it was run. So they all cleared “a substantial quality threshold”. The program also required each study to be large enough to detect the effect it was looking for. This matters for my argument later on. These were not tiny studies that missed an effect by chance.

The team sorted the studies by how the authors themselves described their results. Studies with strong results were 40 percentage points more likely to be published than null results. That is roughly 60 in 100 published, against 20 in 100. Strong results were also 60 percentage points more likely to be written up at all. About 65 in 100 null studies never became a paper.

The team summed up the main problem in one short sentence: “Authors do not write up and submit null findings”. Among studies that did get written up, the strength of the result made no clear difference to whether they were published. I find this more uncomfortable than blaming unfair journals. It means researchers like me cause part of the problem.

What does it cost in medicine?

In clinical research, an unpublished null result wastes more than time and money. It wastes the effort of patients who took on risk so that others could learn.

A 2013 study in the BMJ looked at 585 large trials [4]. Each had at least 500 participants, was registered on ClinicalTrials.gov when it started, and was completed before 2009. Of those, 171 trials, or 29%, stayed unpublished. That is more than 1 in 4. Those unpublished trials had an estimated 299,763 participants.

For 78% of the unpublished trials, no results were posted on the registry either. The authors spelled out the cost. Many participants, they wrote, “were exposed to the risks of trial participation without the societal benefits that accompany the dissemination of trial results”.

Those figures do not tell us how many of the missing trials were negative. But suppose a treatment fails in a trial and nobody hears about it. The next team may test the same idea, recruit new patients, and learn the same lesson again.

A published negative trial would have let the second team spend its money and its patients’ goodwill on a better question.

Is every negative result useful?

I want to be careful here, because not every negative result is useful. A negative result only tells us something if the study could have found a positive one.

A useful negative result usually comes from a study that was planned properly. It has a clear main question set out in advance. It has enough participants to detect an effect of a size that would matter. And its methods are good enough that the answer can be trusted. If a trial like that finds no benefit, other researchers learn something real.

An underpowered study is different. That is a study too small to reliably spot a real effect. It can miss a real effect simply by chance. When it finds no difference, that only means the study could not tell either way.

Publishing it is still better than hiding it, since its data can be pooled with others later. But it should be described honestly as inconclusive, meaning it gave no clear answer either way. It should not be sold as proof that something does not work.

The Nature Communications call named primary endpoints and kept normal peer review. In my view, judging a negative trial against the question it set out to answer is the right test. It separates a real negative finding from a study that was too weak to answer its question.

What can researchers and readers do?

If you run experiments or trials, plan them so that a negative answer would mean something. Then write it up when it comes.

If you review or edit papers, judge a negative paper on its design, not its outcome.

Bottom line

The evidence that results go missing is strong, and much of the loss happens before a paper ever reaches a journal. A journal inviting negative trials is a welcome step. But researchers still have to write those trials up, and readers should judge them by how well the study was designed.

If you read science news, be a little wary of a field where almost every study seems to work. Some answers may already sit in studies that nobody wrote up.

References

[1] Nature Communications, “Clinical trials 2024,” collection call for papers, Dec. 11, 2023. [Online]. Available: https://www.nature.com/collections/idhgjdjegh/about-this-collection

[2] D. Fanelli, “Negative results are disappearing from most disciplines and countries,” Scientometrics, vol. 90, no. 3, pp. 891-904, Mar. 2012, doi: 10.1007/s11192-011-0494-7.

[3] A. Franco, N. Malhotra, and G. Simonovits, “Publication bias in the social sciences: Unlocking the file drawer,” Science, vol. 345, no. 6203, pp. 1502-1505, Sep. 2014, doi: 10.1126/science.1255484.

[4] C. W. Jones, L. Handler, K. E. Crowell, L. G. Keil, M. A. Weaver, and T. F. Platts-Mills, “Non-publication of large randomized clinical trials: cross sectional analysis,” BMJ, vol. 347, Art. no. f6104, Oct. 2013, doi: 10.1136/bmj.f6104.