In August 2023, Kapoor and Narayanan published a survey online in the journal Patterns [1]. It was about a problem called data leakage. They gathered 22 earlier reviews from 17 fields. Each review pointed out errors in research that uses machine learning (ML), where computers learn patterns from data. Leakage showed up in every single one.

The word cheating in my title is a metaphor. A leaky model has no intent. It simply uses information it should not have, and looks smarter than it is.

I work on AI for medical imaging. In my field, the most common form of leakage happens when images from the same patient end up in both the training data and the test data. This usually comes from how research data are handled, not from anyone trying to cheat. I think splitting data by patient should be the default, and every paper should state clearly that it did so.

What happens when one patient is on both sides of the test?

Imagine a teacher who writes the final exam by reusing problems from last month’s homework, word for word. A student who memorized the homework will score very well. The high grade shows the student remembered the homework. It does not show whether the student can solve a new problem.

Kapoor and Narayanan give a more formal definition [1]. Leakage is a false link between what a model sees and the answer it must give. In other words, the model picks up a clue that would not be there in real use. It comes from how the data were collected, sampled or prepared. It usually makes the model look better than it is.

Medical imaging has a similar problem. A single MRI scan is a stack of thin slices. Slices next to each other from the same person look almost the same. Many models learn from these slices one at a time.

Suppose you shuffle all the slices from all your patients. You put most into training and the rest into testing. Then almost every patient ends up on both sides. The model can do well by recognizing the patient instead of the disease.

Yagis and colleagues measured how much this matters in brain MRI [2]. Their models sorted Alzheimer’s and Parkinson’s disease from flat, 2D slices. When they split the data by slice instead of by patient, test accuracy rose by 29 to 55 percentage points, depending on the dataset.

Their most revealing test gave the scans random labels, so there was nothing real to learn. When they split by slice, the network still reached about 96% accuracy. When they split by patient, it scored 50%, the same as a coin flip.

Is leakage a sign of careless researchers?

Some people might blame careless researchers. I think that is unfair. In machine learning contests, the organizers keep the test data hidden from the teams. In most scientific work, the researcher has the whole dataset in hand while building the model, as Kapoor and Narayanan point out [1]. Leakage often happens because the researcher can see all of the data.

Imaging data also tend to arrive as one file per image or slice. So the easiest way to split the data, file by file, is the wrong way.

Sometimes the problem is built into the data. Roberts led a large review of COVID-19 imaging models [3]. It found public datasets that were built by combining other datasets and then shared under new names. The authors warned that this caused a problem. Models were trained and tested on the same or overlapping images. But the people who built the models believed the sources were separate.

I deal with this question in my own work. I build deep-learning models (a common type of AI) for medical imaging. So I always think carefully about how data are split and checked.

Is splitting by patient enough?

Kapoor and Narayanan point to a pneumonia study [1]. Models trained on images from one hospital did not carry over to another hospital. The reason was small differences in how each hospital produced its images. They count this as a broader kind of leakage. Here, the test data do not match the real setting where the model is supposed to work.

The COVID-19 review found an even clearer case. It examined 62 studies in detail [3]. In 16 of them, the comparison group came from a pneumonia dataset of young children. The COVID-19 cases, however, were adults. A model could succeed just by telling children from adults.

Overall, the review judged that none of the 62 models were of potential use in patient care. The reasons were flaws in methods or underlying biases.

So splitting by patient is only the first step. The next question is whether the test data look like the patients the tool will meet.

What does the evidence not show?

The fairest objections come from the researchers themselves. First, the survey’s main count does not show how common leakage is. Kapoor and Narayanan describe their findings as “a lower bound of reproducibility issues in ML-based science” [1]. In other words, they think the real number of problems is likely higher. They built their count from other people’s reviews, not from a random sample of studies.

Those reviews flagged the papers, and Kapoor and Narayanan did not check them again. So a flagged paper is not always a wrong paper. And the survey cannot tell you what share of ML studies are affected.

Second, how much leakage changes the results varies. Kapoor and Narayanan say leakage usually makes a model look better than it is, but not always. Yagis and colleagues also name a limit of their own study [2]. Their patient-level models showed a lot of overfitting, meaning they fit the training data too closely. They also saw the slice-splitting problem mostly in papers that were not peer reviewed, or not reviewed strictly.

I accept these points. My claim is more limited. One common mistake can produce impressive scores even when there is nothing real to learn, as the random-label test shows. And this mistake is cheap to prevent.

What should every imaging AI paper say?

The COVID-19 review already asked for what I want. Its authors asked researchers to state how they kept images from the same patient out of different parts of the data [3]. One way is to describe a patient-level split. This applies even when 3D scans are processed as separate 2D slices.

My view here is also shaped by my training in medical physics. In quality assurance in medical physics, as I see it, we trust a machine only after checking it on measurements that were not used to adjust it. I think an AI model offered as medical evidence should be checked the same way. It should be tested on data it has never seen, from patients it has never seen.

If you are a student, split your data by patient ID before you look at a single image. Then write down how you did it. Some papers do not say that each patient’s images stayed on one side of the split. If you are reading one of those, treat its accuracy as a best case. The real accuracy may be lower.

References

[1] S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine-learning-based science,” Patterns, vol. 4, no. 9, Art. no. 100804, Sep. 2023, doi: 10.1016/j.patter.2023.100804.

[2] E. Yagis, S. W. Atnafu, A. García Seco de Herrera, C. Marzi, R. Scheda, M. Giannelli, et al., “Effect of data leakage in brain MRI classification using 2D convolutional neural networks,” Scientific Reports, vol. 11, Art. no. 22544, Nov. 2021, doi: 10.1038/s41598-021-01681-w.

[3] M. Roberts, D. Driggs, M. Thorpe, J. Gilbey, M. Yeung, S. Ursprung, et al., “Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans,” Nature Machine Intelligence, vol. 3, no. 3, pp. 199-217, Mar. 2021, doi: 10.1038/s42256-021-00307-0.