On April 16, 2024, The BMJ published TRIPOD+AI [1]. It is an updated guideline that tells researchers how to report clinical prediction models. These are tools that estimate a patient’s risk. Most of these tools estimate one of two things. One is how likely it is that a patient has a condition now. The other is how likely it is that the patient will have a certain outcome later. The new version covers models built with a classic statistical method called regression and with machine learning alike. It replaces the 2015 checklist, which the authors say “should no longer be used”.

A reporting guideline may not sound exciting. In my view, though, a performance number cannot be checked unless the paper explains what was done and which patients were involved. The paper also needs to say which data were used, how the data were split and how gaps in the data were handled. Readers should not have to accept “the model performed well” on trust without these details.

How many choices hide behind one number?

I build deep-learning models for breast imaging myself. I know how many decisions go into a single performance number. Someone decides who is included and how the images are prepared. Someone chooses which cases are held back for testing and how the settings are tuned. Someone also decides what happens to incomplete records.

Each of those choices can change the final number, sometimes a lot. Yet the number itself does not show any of them.

The TRIPOD+AI authors make this point [1]. They write that incomplete or inaccurate reporting keeps readers from judging a study’s methods and trusting its findings. They say those readers include patients and the general public. They also warn that poor reporting “might also mask flaws in the design, data collection, or conduct of a study”. Those flaws could cause harm if the model were used in care.

A model’s accuracy number is similar. The number may be honest, but it means little until you know how the model was tested.

What does the guideline actually ask for?

The TRIPOD+AI checklist has 27 main items [1]. A few stand out to me as the ones a reader most needs.

Where the data came from. Authors should describe the data sources for building the model and for testing it, separately. They should explain why they used those data and how well the data represent the patients the model is meant for. They should also give the dates the data were collected and the rules for who was eligible.

How the data were split. Authors should explain how they used the data. That includes whether they divided the data into parts for building and for testing. The guideline’s glossary adds a rule. Test data should not overlap with the data used to train the model, tune its settings or choose between versions. If the same patients sit on both sides of that line, the test partly measures whether the model remembers patients it has already seen.

What happened to missing data. Authors should describe how they handled missing values. They should also give reasons for leaving any data out. Real medical records are full of gaps, and quietly dropping incomplete cases can change who the model was really built for.

How the model was built. Authors should spell out every step in building the model, including how they tuned its settings. They should say whether the data and the analysis code are available. They should also report performance with confidence intervals, which show how much the result could vary by chance. That includes results for key subgroups of patients.

I also like one small rule. When a detail is unknown or does not apply, authors should say so clearly rather than leave it out. Silence and “we don’t know” are different answers, and readers deserve to know which one they are getting.

How big was the problem before?

The problem was well documented before the update. In 2022, Andaur Navarro and colleagues published a systematic review [2]. They studied 152 papers that used a type of machine learning called supervised learning to build clinical prediction models. In supervised learning, a model learns from examples where the answer is already known. The team found the papers through a PubMed search covering 2018 and 2019. They scored each paper against the original TRIPOD checklist.

The typical (median) paper reported 38.7% of the items that applied to it. That is fewer than 2 in 5. Only 44 of the 152 studies described how they handled missing data.

Other gaps were worse. Only 6 of the 116 studies where it applied reported the full model. Only 9 of 152 reported the model’s performance completely.

Not everything was missing. Nearly every study said where its data came from. But the details a reader would need to rebuild, check or use the model were usually absent. The review’s authors noted that most of the models “were unavailable for replication, assessment, or clinical application”.

This review has one limit. That review left out studies that used machine learning to read images. So its numbers do not directly describe imaging AI papers. I would not apply them to imaging papers directly. Still, I see no reason to think imaging papers are free of the same problems.

Why is a checklist not a quality score?

There are three fair reasons not to rely too much on reporting guidelines.

The first point comes from the guideline’s own authors [1]. The advice is about “transparently reporting how prediction model research was conducted,” not about how to build a model. They also state that “the checklist is not a quality appraisal tool”. A fully completed checklist can still describe a model that was trained on too few patients. That model might also be tested on the wrong patients and be useless in practice. Complete reporting makes those flaws visible, but it does not remove them.

The second point is the risk of empty box-ticking. Some authors will treat a required checklist as paperwork and write one vague sentence per item. The TRIPOD+AI authors seem aware of the risk. One item asks about involving patients and the public. They describe it as meant to go “beyond a mere tick box exercise”.

The third point is that the checklist may not fit every kind of study. The 2022 review’s authors admitted that some items in the older checklist may suit machine learning studies less well [2]. Some machine learning methods are also built to handle missing values on their own. That may partly explain why so few papers reported on missing data.

I accept all three points. But none of them is a reason for researchers to share less. In my view, they mean readers should treat a completed checklist as the start of a careful review, and never as the end of it.

What should readers and authors do?

If you write these papers, the guideline’s authors recommend using it early in the writing process [1]. I would open it even sooner, while the study is still being planned. Treat each item as a question a doubtful colleague will ask. If your data or code cannot be shared, say why.

If you read these papers, including news stories about them, look past the headline number. Ask who the patients were, and where and when the data were collected. Ask how the test set was kept apart, and what happened to incomplete records. A paper that only tells you the model performed well has told you almost nothing yet.

References

[1] G. S. Collins, K. G. M. Moons, P. Dhiman, R. D. Riley, A. L. Beam, B. Van Calster, et al., “TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods,” BMJ, vol. 385, Art. no. e078378, Apr. 2024, doi: 10.1136/bmj-2023-078378. (Correction published 18 Apr. 2024 updating author affiliations only.)

[2] C. L. Andaur Navarro, J. A. A. Damen, T. Takada, S. W. J. Nijman, P. Dhiman, J. Ma, et al., “Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review,” BMC Medical Research Methodology, vol. 22, no. 1, Art. no. 12, Jan. 2022, doi: 10.1186/s12874-021-01469-6.