In March 2024, Nature published a Perspective by Lisa Messeri, an anthropologist at Yale, and M. J. Crockett, a psychologist at Princeton [1]. It is an argument, not an experiment. The authors collected no new data. They looked at how scientists describe the future of artificial intelligence (AI) in research, and asked what could go wrong when we trust those tools as partners.

The abstract puts it in one line: AI “risks introducing a phase of scientific enquiry in which we produce more but understand less” [1].

I think this is one of the most useful warnings about AI in science, because it is not mainly about AI being wrong. It is about us feeling right. A fluent explanation or a clean analysis can give you the feeling of understanding without the substance, and the substance lives in the assumptions behind the answer.

Four jobs scientists want AI to do

The argument starts with a map of scientists’ hopes for AI, built from recent publications [1]. The authors call these hopes visions and name four: “Oracle, Surrogate, Quant and Arbiter” [1].

Each vision sits at a different stage of research [1]. The Oracle works at study design, where it searches and summarizes the scientific literature and suggests new hypotheses. The Surrogate works at data collection, generating stand-in data, even for human participants. The Quant works at data analysis, handling datasets too large or complex for people. The Arbiter works at peer review, judging scientific merit and whether findings will replicate.

What these roles share, the authors argue, is a promise to make science more productive, by overcoming our limited time and attention, and more objective, by overcoming our bias [1]. The catch is what happens to our sense of understanding when we hand over those jobs.

Three illusions, in the authors’ words

The paper names three illusions. The first is the illusion of explanatory depth, in which “someone incorrectly believes they have a deeper or more comprehensive level of understanding” than they really have [1]. The second is the illusion of exploratory breadth, where “scientists falsely believe they are exploring the full space of testable hypotheses, whereas they are actually exploring a narrower space of hypotheses testable using AI tools” [1]. The third is the illusion of objectivity, the belief that AI tools have no point of view or can represent every point of view, when in fact they “embed the standpoints of their training data and developers” [1].

These illusions matter because they make a bigger problem harder to see, which the authors call scientific monocultures. They borrow the idea from farming: growing one crop in a field “improves efficiency but makes the crop more vulnerable to pests and disease” [1]. In science, a monoculture forms when some methods, questions and viewpoints “come to dominate alternative approaches” [1].

Part of the problem is the shape of the answers AI tends to give. The authors point out that “although reductive and quantitative explanations tend to produce feelings of understanding, such feelings are not always correlated with actual understanding” [1]. A single accuracy number is easy to mistake for an explanation. Yet, as they note, “even the most accurate predictive model may bear little relation to the actual data-generating process” [1].

That gap, between repeating an answer and knowing what it rests on, is the heart of the paper.

An old illusion with a new trigger

The first illusion is not new. The Perspective builds on a 2002 paper by Leonid Rozenblit and Frank Keil, who reported that “people feel they understand complex phenomena with far greater precision, coherence, and depth than they really do” [2]. They found this overconfidence was strongest for explanations, compared with facts, procedures or stories [2].

In their first study, sixteen Yale graduate students rated how well they understood everyday things such as a zipper, a flush toilet and a helicopter [2]. After writing step-by-step explanations, answering a diagnostic question and reading an expert explanation, nearly all of them lowered their ratings [2].

The illusion shrinks when people try to explain. AI can take that act off our hands. When a tool writes the explanation for you, you never hit the moment where your own understanding runs out, so the feeling of knowing stays intact.

The strongest case against the worry

This paper is a Perspective, so it offers no measurements of how often scientists fall for these illusions or how much harm follows. The authors say themselves that their list of visions “is not an exhaustive list” [1].

They are also not against AI. They write: “To be clear, we do not take the position that AI should never be used in scientific research” [1]. They even suggest that risks may be low for “routine tasks (such as composing emails) or tasks within one’s domain of expertise” [1]. I agree. Treating every chatbot session as a threat to science would be silly.

Still, two points hold up. When AI is used outside a person’s own field, the authors argue, users can “lack the expertise to know when the results are too good to be true” [1]. And they warn that monoculture risks remain “even when AI tools are being implemented by competent users” [1], because the speed AI offers can let AI-led science, and the narrowing that comes with it, spread.

Habits that keep understanding honest

The Perspective asks researchers to be clear about why they are using AI [1]. A Nature editorial published the same week turned this into advice: map your use to one of the four visions, and think about which trap you are most likely to fall into. It also passed on Crockett’s point that using AI to save time on work your team already knows how to do is less risky than using it for expertise you lack.

I would add a few habits. Before accepting an AI answer, try to explain it yourself, without looking, as Rozenblit and Keil’s participants had to. Write down what the tool assumed: what data it saw, what it left out, and what a number like accuracy does and does not measure. Ask whether the question you are studying is the one you wanted, or only the one the tool makes easy. And for anything that matters, have someone with different training look at it, which is close to the authors’ own suggestion of diverse teams [1].

I am an AI researcher who builds deep-learning models for medical imaging and large language model tools that pull structured information out of clinical records, so the gap between a clean output and the assumptions behind it is a practical concern in work like mine.

AI will keep producing explanations that read well, which makes explaining in your own words more useful, not less. Productivity is easy to see and count. Understanding is quieter, and it is the part we cannot afford to hand over.

References

[1] L. Messeri and M. J. Crockett, “Artificial intelligence and illusions of understanding in scientific research,” Nature, vol. 627, no. 8002, pp. 49-58, Mar. 2024, doi: 10.1038/s41586-024-07146-0.

[2] L. Rozenblit and F. Keil, “The misunderstood limits of folk science: An illusion of explanatory depth,” Cognitive Science, vol. 26, no. 5, pp. 521-562, Sep. 2002, doi: 10.1207/s15516709cog2605_1.