In February 2025, the journal The Lancet Digital Health published screening performance results from the full Mammography Screening with Artificial Intelligence (MASAI) trial [1]. The trial ran at four screening sites in southwest Sweden. There, 105,934 women were randomly assigned to one of two groups. One group had screening supported by artificial intelligence (AI). The other had the usual standard, with two radiologists reading every mammogram.

Each group had about 53,000 women. The AI group found 338 cancers, compared with 262 in the standard group. That is 6.4 cancers found per 1,000 women, compared with 5.0, or about 1 extra cancer for every 700 women screened.

Those numbers will get the headlines. What interests me more is how they were produced. This was not a model scoring well on a stored set of old images. It was a randomized trial inside a working national screening program, with real women, real radiologists and real follow-up.

In my view, this kind of evidence, a randomized trial with real patients in a working screening program, is what should make people trust medical AI rather than just find it promising. I think we should expect trials like this far more often.

What can a benchmark tell you, and what can it not?

Most medical AI is judged the same way. Researchers collect a batch of past images where the answer is already known. They hold some back and see how often the model gets them right. That is a benchmark. It is a good first step, because it tells you whether the model learned something real.

A benchmark does not tell you what happens when the model is switched on in a clinic. The images were taken in the past, and nobody’s care changed because of the answer. So a benchmark cannot show whether doctors trust the tool too much or too little. It cannot show whether the tool slows the work down, or whether patients end up better off.

A 2020 review in the BMJ by Nagendran and colleagues looked at studies of deep learning [2]. Deep learning is a common type of AI that learns patterns from many examples. These studies compared AI with clinicians in medical imaging. The review found only 10 records of randomized trials. Only two of them had been published.

It also found 81 studies that were not randomized. Only nine were prospective, meaning they followed patients forward in time. Just six were tested in a real clinical setting. Yet 61 of those 81 said in their summary that the AI was at least as good as clinicians, or better.

Clinical trials are how we learn how a tool works in real use. The review’s own conclusion was critical of the studies that were not trials. Most studies that were not randomized “are not prospective, are at high risk of bias, and deviate from existing reporting standards”.

How was MASAI built differently?

The MASAI team said plainly why they ran a trial. Earlier studies had looked back at past cases. They had shown promising results for AI in mammography, “however, to our knowledge, a randomised trial has not yet been conducted” [3].

The trial included careful steps that a benchmark does not need. Women were randomly assigned, so the two groups were alike from the start. The AI was one commercial product in one version [1]. It sorted exams to be read by either one radiologist or two. Only about 7 in 100 exams, the ones it rated highest risk, went to two readers. It also marked areas that looked suspicious. Radiologists, not the AI, made every final decision to call a woman back. The trial also had a planned safety check, so it could be stopped early if AI screening looked unsafe.

The results of that safety check were published in The Lancet Oncology in August 2023. After 80,033 women had joined, the team ran a safety check they had planned in advance [3]. They had also set, in advance, the lowest rate of cancer detection they would accept: more than 3 cancers per 1,000 women. AI-supported screening passed that test, finding 6.1 cancers per 1,000 women. So the trial was not stopped, and it went on to answer its main question.

Our earlier piece on the human in the loop looked at what those early numbers mean for how radiologists react to AI. In this piece, I care about something else, which is the order of steps. The team checked safety early. They kept going only because it was safe. Then they used the full group of women to answer the main question.

What did the trial measure, and what did it miss?

Cancer detection went up with AI [1]. The AI group found about 29% more cancers (a ratio of 1.29), or 76 more cancers among about 53,000 women. The extra invasive cancers (cancers that had started to grow into nearby tissue) were mostly small and had not spread to the lymph nodes. AI also found 23 more very early cancers that had not grown beyond where they started, called in situ cancers. Most were in the milk ducts. About half of the extra duct cancers were the more aggressive, high-grade kind, and there was no rise in the low-grade kind.

Workload went down. The AI group needed 61,248 screen readings, compared with 109,692 in the standard group, about 44% fewer. Recalls and false positives were not significantly higher with AI. A false positive is a call-back that turns out not to be cancer. About 2.1% of women in the AI group were called back for more tests, compared with 1.9%, a gap that could be due to chance. More of those call-backs also turned out to be cancer. About 3 in 10 women called back in the AI group had cancer, compared with about 1 in 4. The authors wrote that AI “contributes to the early detection of clinically relevant breast cancer and reduces screen-reading workload without increasing false positives”.

The February 2025 paper did not report the trial’s main outcome. That outcome was the interval cancer rate [3]. Interval cancers are cancers that show up between screening rounds. They are a key sign of what screening missed. Measuring them needed two years of follow-up after women joined.

The interval cancer results were published in The Lancet in January 2026. They showed 1.55 such cancers per 1,000 women in the AI group, compared with 1.76 in the standard group. That met the trial’s goal of showing AI was not worse than standard double reading. The AI group had slightly fewer such cancers, but the gap was too small to show that AI was better.

One important outcome is missing. Deaths from breast cancer were not among the outcomes listed [1]. Finding more small cancers is good news only if it leads to fewer women dying. In any screening program, some extra cancers found may be ones that would never have caused harm. The authors note that about half of the extra early duct cancers were of a middle grade. Some of those may be cancers that would never have caused harm, though the numbers were small. MASAI cannot tell us how many.

Is there a fair case against waiting for trials?

Trials have real costs. MASAI randomly assigned more than 100,000 women over about 20 months [1]. Its main result needed two more years of follow-up after that [3]. AI products are updated faster than that. So the version tested may be out of date by the time the results come out.

There is also the question of where the trial ran. In this trial, standard screening meant double reading, with two radiologists reading each mammogram [1]. That was what the AI was compared against. The authors list their own limits too: one Swedish program, one brand of mammography machine, one AI product and low recall rates to begin with. A health system with a single reader, a different AI product or a different group of patients might see different results. One well run trial in one region of one country is strong evidence for that setting. It is weaker evidence for everywhere else.

I accept both points. In my view, they are reasons to run trials in better ways, with built-in safety checks like MASAI’s and more trials in more places. They are not reasons to treat benchmark scores as enough proof.

What would I ask of the next AI tool?

I build and test deep-learning models for breast imaging myself. This work shows me what a set of test images can tell you about a model. It also shows me what those images cannot tell you about patients.

So when I read about a new medical AI tool, I ask what evidence stands behind it. A high score on old images means the idea is worth testing. A study that follows patients forward in a real clinic means it worked in real practice. A randomized trial with outcomes chosen in advance means we can finally compare it fairly with what we already do.

You might be a patient, a doctor or a hospital buyer. Either way, the question to ask is the same. Was this tested on people, in a setting like ours, against the care we give now? If the honest answer is “only on old images,” the tool may still be promising. It just has not earned trust yet.

Bottom line

MASAI shows what good evidence for medical AI looks like: a randomized trial inside a real screening program, with a safety check planned in advance. AI support found more cancers and cut radiologists’ reading by almost half, without a significant rise in false alarms.

It did not show whether AI saves lives, and it covered one setting in one country. In my view, these limits mean we need more trials like it in more places, instead of relying on benchmark scores.

References

[1] V. Hernström, V. Josefsson, H. Sartor, D. Schmidt, A.-M. Larsson, S. Hofvind, et al., “Screening performance and characteristics of breast cancer detected in the Mammography Screening with Artificial Intelligence trial (MASAI): A randomised, controlled, parallel-group, non-inferiority, single-blinded, screening accuracy study,” The Lancet Digital Health, vol. 7, no. 3, pp. e175-e183, Mar. 2025, doi: 10.1016/S2589-7500(24)00267-X.

[2] M. Nagendran, Y. Chen, C. A. Lovejoy, A. C. Gordon, M. Komorowski, H. Harvey, et al., “Artificial intelligence versus clinicians: Systematic review of design, reporting standards, and claims of deep learning studies,” BMJ, vol. 368, Art. no. m689, Mar. 2020, doi: 10.1136/bmj.m689.

[3] K. Lång, V. Josefsson, A.-M. Larsson, S. Larsson, C. Högberg, H. Sartor, et al., “Artificial intelligence-supported screen reading versus standard double reading in the Mammography Screening with Artificial Intelligence trial (MASAI): A clinical safety analysis of a randomised, controlled, non-inferiority, single-blinded, screening accuracy study,” The Lancet Oncology, vol. 24, no. 8, pp. 936-944, Aug. 2023, doi: 10.1016/S1470-2045(23)00298-X.