In July 2024, the journal Nature published a paper with a blunt title: “AI models collapse when trained on recursively generated data” [1]. Shumailov and colleagues asked a simple question. Suppose a large share of the text online was written by earlier AI language models. What happens to future models trained on that text?

Their answer was that training on text written by models, without care, causes lasting damage [1]. And the first thing to go is the rare material at the edges [1].

That paper was about text on the internet in general. I want to ask a narrower question about science. Research papers are some of the most valuable text we have. AI tools already help write a growing share of them.

My view is that nobody has shown the scientific record collapsing, and I am not claiming it is. But the study gives us a good reason to act now. We should start labeling AI help in papers. We should also protect the human-written record while it is still easy to find.

What did the experiment actually do?

The setup is easy to picture. The team took a small language model called OPT-125m, released by Meta [1]. They trained it further on a standard set of text called wikitext2 [1]. Then they had that model write a new dataset. They trained the next model on the machine-written text. They repeated the cycle, generation after generation [1].

In one example, the starting text was a passage about church towers from the Middle Ages. By the ninth generation, the model was writing about “black @-@ tailed jackrabbits, white @-@ tailed jackrabbits, blue @-@ tailed jackrabbits” [1]. It went on to list jackrabbits in other colors too [1]. It had drifted into a repeating loop that had nothing to do with buildings.

The authors describe two stages [1]. In early collapse, the model starts losing information about the tails of the data. These are the unusual, unlikely cases. In late collapse, the model produces something that looks little like the original [1]. Its output often has far less variety [1].

The rare stuff goes first

The key detail is why the tails vanish. Each time a model writes a new dataset, it draws samples from what it learned [1]. Rare events are, by definition, unlikely to be drawn [1]. Miss them once, and the next model never sees them. Over generations, the unusual gets thinner and the common gets louder.

The authors point out why that matters. Unlikely events are often the ones that concern marginalized groups, people on the margins of society [1]. They are also vital for understanding complex systems [1]. A model that forgets them can still sound fluent. It just becomes quietly worse at exactly the cases that are hardest to get right.

That is the part of the paper I keep coming back to. In science, the rare case is often the interesting one. It might be the odd side effect, the unusual patient, or the result that does not fit the pattern.

Science is already writing with AI

This would just be a thought experiment if scientists were not using these tools. They are.

In April 2024, Liang and colleagues posted a preprint, a study shared online before other scientists had checked it [2]. They looked at 950,965 papers on arXiv, bioRxiv and in Nature journals [2]. They estimated how much of the abstracts and introductions had been changed by large language models [2]. They looked across the whole collection. They did not accuse individual papers [2].

For computer science abstracts on arXiv, the estimate reached 17.5% by February 2024 [2]. That is roughly 1 in 6 sentences, estimated across all those abstracts together, not paper by paper [2]. In November 2022, just before ChatGPT launched, it was 2.3% [2]. The authors note that this early level matches their method’s false positive rate, meaning how often it flags text by mistake [2]. For Nature portfolio journals, the estimate was lower, at 6.3% [2].

A changed sentence is not a made-up finding. The estimate counts sentences that a model substantially changed [2]. The authors report that simple AI proofreading only slightly raises it [2]. The method also cannot tell you which paper used AI [2].

Still, the direction is clear. The authors also raise a separate worry. Most of that editing might come from a model owned by one private company. If so, there could be risks to “the security and independence of scientific practice” [2].

Put the two papers side by side, and a worry about the future appears. Future AI models will be trained on scientific text. More of that text now carries some machine-written phrasing. Later models might learn from earlier models’ versions of science. If they do, rare findings and unusual wording could be the first things to fade.

To be clear, this is my own projection. Nobody has shown it happening to the research literature.

Collapse is not inevitable

The strongest reply to this worry comes from the same research field. In another April 2024 preprint, Gerstgrasser and colleagues made a key point [3]. Earlier collapse studies mostly assumed that new data replace old data over time [3]. They argued that in the real world, data more likely pile up [3].

So they kept the original real data. Then they added each new round of AI-made data on top [3]. Collapse did not happen, across a range of model sizes and designs [3].

Shumailov’s own results point the same way. When 10% of the original human data was kept at every generation, the damage was only minor [1]. Their paper also says outright that complete collapse “may or may not occur” depending on the problem [1].

I think this counterpoint is right, and it changes the question. The danger is losing track of the real data. AI-made text alone does not cause that. Collapse needs the human record to be replaced or buried.

The fix sounds simple: keep the originals, and know which is which. But the Gerstgrasser language tests used small models on a simple dataset of children’s stories [3]. So it is still an open question how well the result carries over to technical writing.

Keep the human record findable

Shumailov and colleagues end with a practical point [1]. To keep learning over time, we need access to the original data. We also need data that language models did not create [1]. They add that it is unclear how AI-made content can be tracked on a large scale [1]. They suggest the groups building and releasing these models work together to track where content came from [1].

For science, that points to three habits. First, journals and preprint servers could ask authors to say plainly how they used AI. They could ask for this in a consistent form that computers can read.

Second, archives could keep clearly dated versions of the research literature from before these tools were common. Third, people building models on scientific text could keep real papers in the mix. That means not swapping them out for cheaper summaries written by AI.

This matters to me directly. I build AI tools for medical data, so where training data comes from is a practical question in my own work.

If you write papers, say how you used AI, even when it only tidied your sentences. If you build models, keep track of where your data came from, and keep real data in the training mix. This asks for the kind of record keeping science already values: knowing where the evidence came from.

References

[1] I. Shumailov, Z. Shumaylov, Y. Zhao, N. Papernot, R. Anderson, and Y. Gal, “AI models collapse when trained on recursively generated data,” Nature, vol. 631, no. 8022, pp. 755-759, Jul. 2024, doi: 10.1038/s41586-024-07566-y.

[2] W. Liang, Y. Zhang, Z. Wu, H. Lepp, W. Ji, X. Zhao, et al., “Mapping the increasing use of LLMs in scientific papers,” arXiv:2404.01268, Apr. 2024, doi: 10.48550/arXiv.2404.01268.

[3] M. Gerstgrasser, R. Schaeffer, A. Dey, R. Rafailov, H. Sleight, J. Hughes, et al., “Is model collapse inevitable? Breaking the curse of recursion by accumulating real and synthetic data,” arXiv:2404.01413, Apr. 2024, doi: 10.48550/arXiv.2404.01413.