In August 2023, Kapoor and Narayanan published a survey online in the journal Patterns about a problem called data leakage. They gathered 22 earlier reviews from 17 fields, each pointing out errors in research that uses machine learning (ML), and found that leakage showed up in every single one [1].
The word cheating in my title is a metaphor. A leaky model has no intent. It simply uses information it should not have, and looks smarter than it is.
I work on AI for medical imaging, and my view is this: the most ordinary form of leakage in my field is letting images from the same patient land on both sides of the test. It comes from how research data are handled, not from bad faith. So splitting data by patient should be the default, stated plainly in every paper.
One patient on both sides of the test
Imagine a teacher who writes the final exam by reusing problems from last month’s homework, word for word. A student who memorized the homework will score very well. The grade is real, but it tells you about memory, not about whether the student can solve a new problem.
Kapoor and Narayanan define leakage more formally: a false link between what a model sees and the answer it is asked to give, created by how the data were collected, sampled or prepared, and it usually makes the model look better than it is [1].
Medical imaging has its own version of the homework problem. A single MRI scan is a stack of thin slices, and neighbouring slices from the same person look almost identical. Many models learn from these slices one at a time.
Shuffle all the slices from all your patients, put most into training and the rest into testing, and almost every patient ends up on both sides. The model can then do well by recognizing the patient instead of the disease.
Imagine a patient, call her Maria, whose MRI scan is cut into 60 slices. About fifty land in training and ten in testing. When the model sees one of those ten slices, it has already studied the slice right next to it, from the same body on the same day. It gets the answer right, but it may simply remember Maria.
Yagis and colleagues measured how much this matters in brain MRI, using models that classify Alzheimer’s and Parkinson’s disease from 2D slices. When they split the data by slice instead of by patient, test accuracy rose by 29 to 55 percentage points, depending on the dataset [2]. In their most telling test, they assigned the labels at random, so there was nothing real to learn. With a slice-level split, the network still reached about 96% accuracy. With a patient-level split it scored 50%, which is what a coin flip gives you [2].
A structural hazard, not a character flaw
It would be easy to blame careless researchers. I think that is wrong. In machine learning competitions, the organizers keep the test set locked away. In most scientific work, Kapoor and Narayanan point out, the researcher has the entire dataset in hand while building the model, and leakage often happens precisely because of that access [1]. Imaging data also tend to arrive as one file per image or slice, so the easiest split is the wrong one.
Imagine a graduate student handed a folder of 20,000 image files, one per slice. A tutorial she found says to shuffle the files and set aside a fifth for testing. She follows it exactly, and her model scores beautifully. She has done nothing dishonest; she simply split files instead of people.
Sometimes the problem is built into the data. A large review of COVID-19 imaging models, led by Roberts, described public datasets stitched together from other datasets and shared under new names. The authors warned that this repackaging led models to be trained and tested on the same or overlapping images while their developers believed the sources were separate [3].
This is a question I deal with directly. I lead BRACE, a deep-learning project that uses breast MRI to tell inflammatory breast cancer apart from other locally advanced breast cancers. I built its model-training and validation pipelines and am now extending it through external validation, so how data are split and checked is never an afterthought for me.
Splitting by patient is necessary, not sufficient
Kapoor and Narayanan point to a pneumonia study in which models trained on images from one hospital did not carry over to another hospital, because of subtle differences in how each hospital produced its images [1]. They count this under a broader kind of leakage, where the test data do not represent the situation the claim is about.
The COVID-19 review found a starker case. In 16 of the 62 studies it examined in detail, the comparison group came from a pneumonia dataset of young children, while the COVID-19 cases were adults [3]. A model could succeed by telling children from adults. Overall, the review judged that none of the 62 models were of potential clinical use, because of flaws in methods or underlying biases [3].
So the patient-level split is the floor, not the ceiling. After it comes the question of whether the test data look like the patients the tool will meet.
What the evidence does not show
The fairest objections come from the researchers themselves. First, the headline count is not a rate. Kapoor and Narayanan describe their findings as “a lower bound of reproducibility issues in ML-based science,” compiled from other people’s reviews rather than a systematic sample [1]. The tallied papers were flagged by those reviews, not re-checked, so a flagged paper is not automatically a wrong paper, and the survey cannot tell you what share of ML studies are affected.
Second, the size of the effect varies. Kapoor and Narayanan say leakage usually inflates performance, which is not the same as always [1]. Yagis and colleagues also flag the substantial overfitting in their patient-level models as a limitation of their own study [2]. They also saw the slice-splitting problem mostly in manuscripts that were not peer reviewed, or not rigorously so [2].
I accept all of this. My claim is narrower: one common mistake can produce impressive numbers out of nothing, as the random-label test shows, and it is cheap to prevent.
What I would ask of every imaging AI paper
The COVID-19 review already made my request. Its authors asked researchers to state how they kept images from the same patient out of different parts of the data, for example by describing a patient-level split, including when 3D scans are processed as separate 2D slices [3].
My view here is also shaped by the field I am applying to train in. The way I see quality assurance in medical physics, a system earns trust when it is checked on measurements that were not used to tune it. I think an AI model offered as clinical evidence deserves the same habit: verify it on data it has never seen, from patients it has never seen.
Most imaging AI is not broken. But one common, cheap-to-prevent mistake can manufacture impressive numbers out of nothing. Split by patient, say so in the paper, and test on patients the model has never seen.
If you are a student, split your data by patient ID before you look at a single image, and write down how you did it. If you are reading a paper that does not say each patient’s images stayed on one side of the split, treat its accuracy as a best case.
References
[1] S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine-learning-based science,” Patterns, vol. 4, no. 9, Art. no. 100804, Sep. 2023, doi: 10.1016/j.patter.2023.100804.
[2] E. Yagis, S. W. Atnafu, A. García Seco de Herrera, C. Marzi, R. Scheda, M. Giannelli, et al., “Effect of data leakage in brain MRI classification using 2D convolutional neural networks,” Scientific Reports, vol. 11, Art. no. 22544, Nov. 2021, doi: 10.1038/s41598-021-01681-w.
[3] M. Roberts, D. Driggs, M. Thorpe, J. Gilbey, M. Yeung, S. Ursprung, et al., “Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans,” Nature Machine Intelligence, vol. 3, no. 3, pp. 199-217, Mar. 2021, doi: 10.1038/s42256-021-00307-0.
Related Articles
