In March 2024, the journal Nature published a Perspective, a kind of expert essay [1]. Its authors were Lisa Messeri, an anthropologist at Yale, and M. J. Crockett, a psychologist at Princeton. It is an argument, not an experiment. The authors collected no new data. They looked at how scientists describe the future of artificial intelligence (AI) in research. Then they asked what could go wrong when we trust those tools as partners.

The paper’s abstract puts it in one line. AI “risks introducing a phase of scientific enquiry in which we produce more but understand less”.

I think this is one of the most useful warnings about AI in science. Its main concern is that AI can make scientists feel they understand something better than they actually do, which is a different risk from AI simply giving wrong answers. A smooth explanation or a clean analysis can make a result feel clear even when we have not looked at the assumptions behind it. In my view, real understanding requires knowing those assumptions.

What four jobs do scientists want AI to do?

The argument starts with a list of what scientists hope AI will do, drawn from recent published work [1]. The authors call these hopes visions and name four: “Oracle, Surrogate, Quant and Arbiter”.

Each vision fits a different stage of research. The Oracle helps design studies. It searches and sums up the scientific literature and suggests new ideas to test. The Surrogate helps collect data by producing stand-in data, even in place of human volunteers. The Quant helps analyze data sets too large or complex for people. The Arbiter helps with peer review, the process where experts check a study before it is published. It judges how good the science is and whether the findings will hold up when repeated.

The authors argue that these roles share a common promise. They promise to make science more productive by getting past our limited time and attention. They also promise to make it more objective by getting past our bias. The authors worry about what happens to our sense of understanding when we hand those jobs to AI.

What are the three illusions?

The paper names three illusions. The first is the illusion of explanatory depth, which means feeling you can explain something better than you really can. In the authors’ words, “someone incorrectly believes they have a deeper or more comprehensive level of understanding” than they really have [1].

The second is the illusion of exploratory breadth. Scientists wrongly believe they are exploring every idea that could be tested. In fact, they are exploring a narrower set: only the ideas that AI tools can test.

The third is the illusion of objectivity. This is the belief that AI tools have no point of view, or can stand for every point of view. In fact, the tools “embed the standpoints of their training data and developers”.

These illusions matter because they hide a bigger problem, which the authors call scientific monocultures. They borrow the idea from farming. Growing one crop in a field “improves efficiency but makes the crop more vulnerable to pests and disease”. In science, a monoculture forms when some methods, questions and viewpoints “come to dominate alternative approaches”.

Part of the problem is the kind of answers AI tends to give. The authors write that “although reductive and quantitative explanations tend to produce feelings of understanding, such feelings are not always correlated with actual understanding”. In plain terms, simple answers built on numbers feel satisfying, but that feeling can mislead. A single accuracy number is easy to mistake for an explanation. Yet, as they note, “even the most accurate predictive model may bear little relation to the actual data-generating process”. In other words, a model can make good predictions without matching how the data really came about.

This gap is the main subject of the paper. Being able to repeat an answer is different from knowing what the answer is based on.

Is this an old illusion with a new trigger?

The first illusion is not new. The Perspective builds on a 2002 paper by Leonid Rozenblit and Frank Keil [2]. They reported that “people feel they understand complex phenomena with far greater precision, coherence, and depth than they really do”. This overconfidence was strongest for explanations, compared with facts, procedures or stories.

In their first study, sixteen Yale graduate students rated how well they understood everyday things. These included a zipper, a flush toilet and a helicopter. Then the students wrote step-by-step explanations, answered a question that tested their knowledge, and read an expert’s explanation. After that, nearly all of them lowered their ratings.

The illusion gets weaker when people try to explain something themselves. AI can do that explaining for us. When a tool writes the explanation for you, you never find out where your own knowledge stops. So you keep feeling that you understand.

What is the strongest case against the worry?

This paper is an essay, so it offers no measurements [1]. It does not show how often scientists fall for these illusions or how much harm follows. The authors say themselves that their list of visions “is not an exhaustive list”.

They are also not against AI. They write: “To be clear, we do not take the position that AI should never be used in scientific research”. They even suggest that risks may be low for “routine tasks (such as composing emails) or tasks within one’s domain of expertise”. I agree. Treating every chatbot session as a threat to science would be silly.

Still, I find two of their points convincing. The first is about using AI outside your own field. In that case, the authors argue, users can “lack the expertise to know when the results are too good to be true”. The second is that monoculture risks remain “even when AI tools are being implemented by competent users”. The reason is speed. Because AI works fast, AI-led science can spread quickly, and the narrowing that comes with it can spread too.

Which habits keep understanding honest?

The Perspective asks researchers to be clear about why they are using AI [1]. A Nature editorial published the same week turned this into advice. It said to match your use of AI to one of the four visions. It also said to think about which trap you are most likely to fall into. And it passed on a point from Crockett. Using AI to save time on work your team already knows how to do is less risky than using it for skills you lack.

I would add a few habits. Before you accept an AI answer, try to explain it yourself without looking, as Rozenblit and Keil’s volunteers had to. Write down what the tool assumed. Note what data it saw, what it left out, and what a number like accuracy does and does not measure.

Ask whether the question you are studying is the one you wanted, or only the one the tool makes easy. For any important result, have someone with different training look at it. That is close to the authors’ own suggestion of diverse teams.

I am an AI researcher who works with medical data. For me, the gap between a clean output and the assumptions behind it is a practical concern, not an abstract one.

AI will keep producing explanations that read well. That makes it even more useful to explain things in your own words. It is easy to see and count how much work gets done. Understanding is harder to see, and I think it is the part of science we cannot afford to hand over to AI.

References

[1] L. Messeri and M. J. Crockett, “Artificial intelligence and illusions of understanding in scientific research,” Nature, vol. 627, no. 8002, pp. 49-58, Mar. 2024, doi: 10.1038/s41586-024-07146-0.

[2] L. Rozenblit and F. Keil, “The misunderstood limits of folk science: An illusion of explanatory depth,” Cognitive Science, vol. 26, no. 5, pp. 521-562, Sep. 2002, doi: 10.1207/s15516709cog2605_1.