In 2024, the journal Radiology: Artificial Intelligence published an update to CLAIM [1]. CLAIM stands for the Checklist for Artificial Intelligence in Medical Imaging. It is not an experiment, and it does not test any model.
Instead, it is a list of things a research paper about imaging AI should tell its readers. A panel of 72 people who finished the process helped write it. They included doctors, AI scientists, journal editors and statisticians. The authors call it “an educational tool for both authors and reviewers”.
The checklist rarely gets public attention, but I think it carries an important message. Most news about medical AI leads with a single number, usually the AUC. A high AUC shows that a model does well at one specific task, but it does not show that the model will help patients. Going through the CLAIM items one by one helps explain why.
What does an AUC actually measure?
AUC stands for area under the curve. The curve is a receiver operating characteristic (ROC) curve. The name is technical, but the idea is simple.
A good model “should give higher risk estimates for patients with the event than for patients without the event” [2]. Statisticians call this skill discrimination. The AUC is the usual way to put a number on it.
In plain words, pick at random one patient who has the disease and one who does not. The AUC is the chance that the model gives the sick patient the higher score. Ties count as half. An AUC of 0.5 is no better than a coin flip. An AUC of 1.0 means the model ranks every sick patient in the test group above every healthy one.
Imagine a teacher who can always tell which of two essays is better, but who gives every essay a grade two letters too high. Her ranking is perfect. But any student who trusted her grades would be misled.
Ranking is a real skill, and a model with a poor AUC is not worth much. But good ranking is only the first thing to check. The rest of this post covers what the AUC leaves out.
Who was in the test?
An AUC is always measured on some particular group of patients. It only describes that group.
CLAIM asks authors to “describe how well the data align with the intended use and target population of the model” [1]. It also asks them to spell out who was included and who was left out. This includes the hospital setting and the patients’ age, sex and race.
In a 2020 review in The BMJ, Nagendran and colleagues looked at studies comparing deep learning with expert doctors on medical images [3]. Deep learning is a common type of AI. Of 81 studies that were not randomized trials, “only nine were prospective and just six were tested in a real world clinical setting”.
Prospective means the researchers planned the study first and then enrolled patients and collected data going forward. They did not pull the data from old records. CLAIM asks authors to “evaluate AI models in a prospective setting, if possible” [1].
A new hospital is a new exam
A model learns from the scanners, patients and habits of the hospitals whose data trained it. A new hospital differs in all of these.
Checking a model on data from another site is called external testing. CLAIM prefers this plain term [1]. It asks authors who skip it to “note and justify this limitation”.
Kelly and colleagues wrote about this in BMC Medicine in 2019 [4]. They argued that judging real-world performance needs testing on outside data. In their words, that means “adequately sized datasets collected from institutions other than those that provided the data for model training”.
They gave an example from one chest X-ray study. Specificity is how often healthy people are correctly told they are healthy. At one fixed cutoff, the model’s specificity ranged from 0.566 to 1.000 across five independent datasets. In other words, it correctly cleared anywhere from about 57 to 100 out of every 100 healthy people. It was the same model with the same cutoff, yet the results were very different depending on where it was used.
The cutoff is the score above which the model flags a patient as sick. It is easy to overlook. The AUC sums up every possible cutoff at once. But in the clinic, someone has to pick one. A cutoff that looked sensible in the first hospital can flag far too many healthy people somewhere else.
Ranking well is not the same as being right about risk
Many models give a probability, such as a 35% chance of cancer. Whether those numbers can be trusted is called calibration.
In 2019, Van Calster and colleagues wrote a paper that called calibration the Achilles heel of predictive analytics [2]. In other words, it is the weak spot of tools that predict risk. They wrote: “estimated risks can be unreliable even when the algorithms have good discrimination”.
One common cause is using the model in a different setting. Suppose a model was built where a disease is common. It “may systematically give overestimated risk estimates when used in a setting where the incidence is lower”. In other words, it may predict too much risk where the disease is rarer.
This is the line from their paper that I would most like readers to remember. Poor calibration “may make an algorithm less clinically useful than a competitor algorithm that has a lower AUC but is well calibrated”. CLAIM reflects this too. It asks authors to use ROC analysis and, where it fits, calibration curves [1]. Calibration curves show whether a model’s predicted risks match what really happened.
Do patients end up better off?
Kelly and colleagues noted that the usual performance measures miss what matters most to patients [4]. What matters is whether using the model actually changes their care for the better.
They described a randomized trial during labor. In it, a computer read the monitor that tracks the unborn baby’s heartbeat. It found no improvement in outcomes for mothers or babies. They called it “a cautionary example of how higher accuracy enabled by AI systems does not necessarily result in better patient outcomes”.
Meanwhile, many study abstracts claimed more than the evidence showed. In the Nagendran review, 61 of 81 studies said in their abstract that the AI was at least comparable to clinicians, or better [3]. Only 31 said more prospective studies or trials were needed.
CLAIM asks authors to describe how their model is meant to be used and its role in care [1]. It also asks them to “discuss any issues that would impede successful translation of the model into practice”.
One fair objection is that a checklist cannot make a study good. CLAIM itself says it was “not designed as a scoring system”. A paper can tick every box and still describe a model nobody needs, and I agree with that. But a checklist can show readers what a paper left out. In my view, once those gaps are visible, a single impressive number cannot easily hide them.
I think about this in my own work on AI for breast imaging. A score measured on your own data only shows how well the model ranks patients there.
A high AUC means a model has learned to rank sick patients above healthy ones. It earns the word success only after it holds up on new patients and gives honest risk estimates at a sensible cutoff. It must also make someone’s care better.
So the next time you read that an AI model achieved an AUC of 0.95, look for four more things in the story. Check who the patients were, and whether they resemble the people the tool is meant for. Look for a test at a hospital that had nothing to do with building it. Look for any word on whether its risk numbers can be trusted, and what cutoff doctors will use. And look for evidence that patients actually did better. If those pieces are missing, the AUC tells only the start of the story. Nobody yet knows whether the tool helps patients.
References
[1] A. S. Tejani, M. E. Klontzas, A. A. Gatti, J. T. Mongan, L. Moy, S. H. Park, et al., “Checklist for Artificial Intelligence in Medical Imaging (CLAIM): 2024 Update,” Radiology: Artificial Intelligence, vol. 6, no. 4, Art. no. e240300, Jul. 2024, doi: 10.1148/ryai.240300.
[2] B. Van Calster, D. J. McLernon, M. van Smeden, L. Wynants, and E. W. Steyerberg, “Calibration: the Achilles heel of predictive analytics,” BMC Medicine, vol. 17, Art. no. 230, Dec. 2019, doi: 10.1186/s12916-019-1466-7.
[3] M. Nagendran, Y. Chen, C. A. Lovejoy, A. C. Gordon, M. Komorowski, H. Harvey, et al., “Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies,” BMJ, Art. no. m689, Mar. 2020, doi: 10.1136/bmj.m689.
[4] C. J. Kelly, A. Karthikesalingam, M. Suleyman, G. Corrado, and D. King, “Key challenges for delivering clinical impact with artificial intelligence,” BMC Medicine, vol. 17, Art. no. 195, Oct. 2019, doi: 10.1186/s12916-019-1426-2.
Related Articles
