In March 2024, Weixin Liang, James Zou and colleagues posted a study [1]. It tried to measure how much peer review text was now being written with help from ChatGPT. Many people in science had only talked about this quietly. Peer review is the step where other experts judge a study before it is accepted.

The team looked at reviews for four large artificial intelligence (AI) conferences held after ChatGPT came out. These were ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023. They estimated that between 6.5% and 16.9% of the review text could have been substantially changed by a large language model (LLM). That means more than spell-checking or small edits. The paper was accepted to the ICML 2024 machine learning conference.

The study cannot tell us who did the thinking in any single review, so I will be careful about what it shows. In my view, using AI to improve the wording of a review is fine, but using it to decide the verdict is not. Pasting someone’s unpublished manuscript into a chatbot is a separate problem.

What did the study measure?

The method does not read a review and decide whether a machine wrote it. It estimates what share of a whole collection of text looks like AI output [1]. It does not judge any single review. Think of it like estimating how much of a city’s water comes from one river. You sample the reservoir instead of testing each tap.

Before ChatGPT, the estimates were low: 1.6% at ICLR, 1.9% at NeurIPS and 2.4% at CoRL. Afterward they rose to 10.6%, 9.1% and 6.5%. EMNLP, a conference on language processing, had no earlier data. But it had the highest figure, about 16.9%. Reviews for Nature Portfolio journals showed no clear increase.

The team also ran reviews through ChatGPT only to fix typos and grammar. The estimate barely moved, so the method is picking up more than proofreading. The authors are clear on one point. They “do not claim (nor do we believe) that many reviewers are using ChatGPT to write entire reviews outright”. A reviewer could jot down bullet points and ask the tool to turn them into paragraphs. That would still count.

So a real, measurable share of review text was reshaped by AI. But it is an estimate across the whole collection. It does not tell you which reviewers did it. It does not show that anyone handed over the whole job.

Where did the AI text show up?

The revealing part is where the AI-shaped text appeared. The estimated share was higher in reviews sent in within three days of the deadline [1]. At ICLR 2024, it was 11.3% close to the deadline, against 8.8% earlier. It was also higher in reviews where reviewers rated their own confidence as 2 or lower on a 5-point scale.

It was lower in reviews that contained et al.. The authors used that phrase as a rough sign that the reviewer cited other research. At ICLR and NeurIPS, it was also higher among reviewers who did not respond to the authors’ rebuttals, meaning the authors’ written replies to the reviews. Even the vocabulary shifted. In ICLR 2024 reviews, commendable became 9.8 times more likely to appear in a sentence. Meticulous became 34.7 times more likely.

All of these show that two things went together. They do not show that one caused the other. The authors say they “cannot make a causal claim” about the discussion effect. They also suggest a more generous explanation. Conferences are short of reviewers, so scholars may agree to take on more reviews and lean on the tool to cope. For citations, they note they lack a counterfactual, meaning a way to see what would have happened without the tool. People who reach for ChatGPT might have cited less even without it.

This pattern worries me. The AI text showed up most in reviews with the fewest signs of careful human attention.

Is polishing words the same as handing off judgment?

I build language model tools myself. I know how good these systems are at turning rough input into clean prose. That is why I think the rule should be about judgment. Help with the wording is fine, but the judgment should stay with the reviewer.

A peer review has two parts. One is the verdict. Is the method sound? Does the evidence support the claim? What is missing? The other part is the writing that carries that verdict to the authors and editors. Asking a tool to tidy the second part is like asking a colleague to check your grammar. Asking it to supply the first part means the paper was never really reviewed by a peer.

The study hints at what is lost when the tool does part of the judging. Reviews most similar to the other reviews of the same paper had more AI-shaped text [1]. The authors warn that authors then “lose an opportunity to receive feedback from multiple, independent, diverse experts in their field”. Peer review works because different people notice different things. If every reviewer runs their notes through the same model, the reviews start to sound alike. The reviews then lose some of their independence.

To be fair, the authors themselves decline to call AI use in reviewing good or bad. Some reviewers write in a second language. For them, a tool that helps them say what they mean is a real benefit. If the thinking is theirs and the tool only shapes the sentences, I have no complaint.

Whose manuscript is it to share?

The study flags a second issue but leaves it aside. It names the “privacy and anonymity risks of providing unpublished work to a privately owned language model” [1]. That risk starts as soon as you paste the work into the tool, no matter what the tool writes back.

The US National Institutes of Health (NIH) acted on this early. In a June 2023 notice, it set a rule for its peer reviewers. They may not use large language models or other generative AI to analyze and write critiques of grant applications [2]. Its reasoning was about confidentiality. Uploading content from an application to online AI tools breaks its review rules. The notice explains that “AI tools have no guarantee of where data are being sent, saved, viewed, or used in the future.”

NIH’s rule covers grant review, not journal or conference manuscripts. But I think the same reasoning applies to them. A manuscript under review is someone else’s unpublished work, shared with you in trust. It is not your material to hand to a third party, however useful that party may be.

For reviewers, that means a few plain habits. Read the paper yourself, and write your own points first. Check any citation you add. Do not paste confidential text into a public chatbot. For editors and conference organizers, it means clear rules on disclosure and on what may be uploaded. It also means review loads that leave time to think. The study cannot tell us who did the thinking in any given review. It does tell us the machine’s share was higher in late, low-confidence reviews. Last-minute, low-confidence reviewing is a problem we can actually fix.

References

[1] W. Liang, Z. Izzo, Y. Zhang, H. Lepp, H. Cao, X. Zhao, et al., “Monitoring AI-modified content at scale: A case study on the impact of ChatGPT on AI conference peer reviews,” arXiv:2403.07183, Mar. 2024, rev. Jun. 2024 (accepted to ICML 2024), doi: 10.48550/arXiv.2403.07183.

[2] National Institutes of Health, “The use of generative artificial intelligence technologies is prohibited for the NIH peer review process,” NIH Guide Notice NOT-OD-23-149, Jun. 23, 2023. [Online]. Available: https://grants.nih.gov/grants/guide/notice-files/NOT-OD-23-149.html