In January 2024, the journal GigaScience published a study that tried something most of us never do: rerun other people’s published code. Sheeba Samuel and Daniel Mietchen searched biomedical papers in PubMed Central for Jupyter notebooks, the interactive documents many researchers use to mix code, notes and results. They found the matching GitHub repositories and tried to rerun the Python notebooks automatically, in a setup as close to the original as they could build [1].

They started with 27,271 notebooks linked to 3,467 papers [1]. In the end, 879 of them ran and gave the same results the authors had published [1].

That number is sobering, but I do not read it as proof that biomedical science is broken. My reading is narrower and, I think, more useful: putting code on GitHub is necessary but nowhere near enough. Whether anyone can run it again depends on things most of us treat as boring housekeeping: which versions of which software we used, where the data lives, and whether the instructions actually work.

What the rerun found

The funnel is the story. Of the 22,578 notebooks written in Python, 15,817 listed their software requirements in standard files, so the pipeline could attempt them [1]. For 10,388, every listed package installed. Of those, 1,203 ran from top to bottom without an error, and 879 of those produced identical results [1]. The other 324 ran fine but gave different answers [1].

Put another way, fewer than 4 in 100 of all the Python notebooks, counting those the pipeline never attempted, ended up reproducing their published results. And these were not random scripts from the internet. They were attached to peer-reviewed papers, and the authors conclude that review at these journals “does not generally pay much attention to the reproducibility of the notebooks” [1].

Most failures were housekeeping, not science

The errors were mostly mundane. The most common were a module that could not be found, a file that could not be found, and a failed import [1]. Missing or broken modules alone stopped 6,588 runs, about 42% of all the notebooks they tried [1].

Versions matter more than people expect. If a notebook named a package without saying which version, the rerun simply installed the newest one available [1]. Software moves on. A function gets renamed, a default changes, and code that ran perfectly in 2019 stops at line three in 2023.

The same goes for notebooks that ran but gave different numbers. Among the notebooks that ran cleanly, many with identical results had declared their dependencies, compared with only one in the group whose results differed [1]. The authors also list other reasons an error-free run can drift: random functions, data that changes over time, or cells that were originally run more than once or out of order [1].

A failed run is not a failed study

This is where I want to be fair to the thousands of researchers behind those notebooks. The study was a deliberately strict test. It used only the first error in each notebook, did not use Docker images even where one may have been available, and made no manual fixes [1]. The authors note that many of the missing software and data problems are often straightforward to fix by hand for a single notebook; doing that for thousands was simply not practical [1].

Most importantly, the study was not designed to test whether the papers’ conclusions hold. The authors say plainly that it “does not address inferential reproducibility” [1]. A notebook that crashes on a missing file tells you the file is missing. It does not tell you the finding is false.

So a headline claiming that most biomedical code is broken, and therefore the science is unreliable, would go further than the data. I think the honest reading is that we rarely give anyone else a fair chance to check our work. That is a problem worth fixing even when the science is sound.

Sharing on request was never enough

None of this is new. In 2018, Victoria Stodden and colleagues took a random sample of 204 papers in Science published after the journal began requiring authors to share data and code on request [2]. They could obtain the materials for 44% of the papers and reproduce the findings for 26% [2]. Their verdict was that the policy was “an improvement over no policy, but currently insufficient for reproducibility” [2].

The FAIR principles, published in 2016, point in the same direction. FAIR stands for findable, accessible, interoperable and reusable, and its authors wanted the principles to cover “the algorithms, tools, and workflows that led to that data,” not only the data themselves [3]. They also put weight on machines being able to find and use data automatically, not only people [3]. A notebook that only runs on one laptop, with data in a folder nobody else can see, fails that test long before any reviewer looks at it.

What I would do before pressing publish

My own work sits right on this problem. I build automated pipelines for imaging and clinical data, and I built the image processing, training and validation pipelines for BRACE, a breast MRI deep-learning project. Pipelines like these are only worth building if they give the same answer the next time someone runs them, including me six months later.

The good news from Samuel and Mietchen is that the fixes they point to are ordinary habits, not new technology. Their data also link documentation with success: notebooks with more written explanation generally did better [1]. They also call for wider use of standard dependency files and for automated checks of dependencies, versions and outputs during publishing [1].

For students and researchers, my short list is this. Pin exact versions of every package, not just their names. If your setup is complicated, provide a container or an environment file that rebuilds it. Say where the data lives and how to get it, even if access needs an application. Set random seeds, and run your notebook from top to bottom in a clean session before you save it. And write a README that someone outside your lab can follow without emailing you.

Journals can help by adding basic automated checks to review, as the authors suggest [1]. But most of this is within our own control, today, on our own projects. A study that someone else can rerun is easier to trust, and much easier to correct when something turns out to be wrong.

References

[1] S. Samuel and D. Mietchen, “Computational reproducibility of Jupyter notebooks from biomedical publications,” GigaScience, vol. 13, Art. no. giad113, Jan. 2024, doi: 10.1093/gigascience/giad113.

[2] V. Stodden, J. Seiler, and Z. Ma, “An empirical analysis of journal policy effectiveness for computational reproducibility,” Proceedings of the National Academy of Sciences, vol. 115, no. 11, pp. 2584-2589, Mar. 2018, doi: 10.1073/pnas.1708290115.

[3] M. D. Wilkinson, M. Dumontier, I. J. Aalbersberg, G. Appleton, M. Axton, A. Baak, et al., “The FAIR Guiding Principles for scientific data management and stewardship,” Scientific Data, vol. 3, Art. no. 160018, Mar. 2016, doi: 10.1038/sdata.2016.18.