In October 2022, a team of economists published an experiment in PNAS with an irresistible design [1]. They took one real, unpublished research paper in finance, written jointly by a Nobel laureate and a relatively unknown early career researcher, and sent it out for review [1]. Some reviewers saw the famous name, some saw the junior name, and some saw no name at all [1].

Same paper, same words, same data. The verdicts were very different.

I read this study as clear evidence that prestige shapes how experts judge work, even when they believe they are judging the work alone. But I do not read it as proof that peer review is rotten, or that knowing who did the research never counts for anything. The more useful lesson is narrower: hiding names helps, and it has limits we should be honest about.

One paper, three names

The prominent author was Vernon L. Smith, who won the 2002 Nobel Memorial Prize in Economic Sciences, and the junior author was Sabiou Inoua, an early career research associate [1]. Both worked at the same institute at Chapman University, which let the researchers hold the affiliation constant [1].

The paper went to the Journal of Behavioral and Experimental Finance, published by Elsevier, and the journal’s editor, who was also one of the study’s authors, sent review invitations to more than 3,300 potential reviewers [1]. The paper had not been posted online or presented anywhere before the data collection was finished, so reviewers could not look up who wrote it [1]. The whole plan was preregistered, meaning the team wrote down its hypotheses and tests before collecting data [1].

The famous name worked before anyone saw the full manuscript. Among the 2,611 researchers who answered the invitation, 38.5% agreed to review when Smith was named, compared with 30.7% when no name was given and 28.5% when Inoua was named [1]. Only the gap for Smith was statistically significant [1].

What the reviewers decided

Reviewers submitted 534 reports in total [1]. For the cleanest comparison, the team used 313 reports from reviewers whose invitation carried no name, so the only thing that differed was the name printed on the manuscript itself [1].

When the manuscript showed Inoua’s name, 65.4% of reviewers recommended rejection. With no name, 48.2% did. With Smith’s name, 22.6% did [1]. At the other end, more than 20% recommended outright acceptance when the Nobel laureate was shown, but fewer than 2% did so for the research associate [1].

The authors think the most plausible explanation is that reviewers, consciously or not, “ascribed higher quality to a paper authored by a prominent researcher” [1]. They compare it to the halo effect from social psychology, where a favorable first impression leads people to rate what they do not yet know more highly too [1]. I find that convincing. Nobody here had to be corrupt. A good reputation simply made the same arguments look stronger.

Not the first warning

A 2017 study in PNAS found a similar pattern in computer science [2]. At the Web Search and Data Mining (WSDM) conference, a selective meeting that accepted 15.6% of submissions, each of the 500 papers was assigned two reviewers who could see author names and affiliations and two who could not [2]. Reviewers who could see the names were significantly more likely to recommend accepting papers from famous authors, top universities and top companies [2].

That is a different field, a different kind of venue, and a different design, which is exactly why it matters to me. Two careful experiments, five years apart, point the same way.

The case for knowing who wrote it

There is a fair argument on the other side, and the 2017 authors state it themselves. In one reading, reviewers who see names “make use of prior information that may allow them to make better overall judgments” [2]. A researcher with a long record of solid work probably does produce solid work more often. Using that information is not irrational.

The trouble, as the same authors put it, is that “two contributions of roughly equal merit might be scored differently” [2]. A reputation is a reasonable guess about the past. Peer review is supposed to judge this particular paper. The Huber experiment is useful because it removes the guessing: the paper was identical, so the large gaps in the verdicts came from the name shown, not the work [1].

The study also has clear limits. It used one paper in one field, finance, and the authors say they “cannot unambiguously speak for other fields of science” [1]. The two authors also differed in name origin and skin tone, which the team lists as another possible source of bias it could not rule out, though it notes that a photo of Inoua was not easy to find online at the time [1]. And they suggest repeating the test with an outstanding paper, a particularly bad paper, and a very high prestige journal instead of the “rather young journal” they used [1].

Hiding names helps, up to a point

The authors conclude that their results support double-anonymized review, where reviewers do not see who wrote the paper [1]. I agree. But they also warn that as more working papers and preprints appear online, “a truly double-anonymized review process becomes less and less realistic” [1].

The 2017 study raised the same problem: some work, such as a large and well-known project, is impossible to anonymize, and early versions often appear on public preprint sites [2]. Reviewers can also go looking for the names on their own [2]. Even in the Huber data, the advantage of full anonymity for the junior author was no longer statistically significant once the team accounted for which reviewers chose to take the job, although the gap for the famous author held [1].

As a researcher, I think the practical answer is to treat anonymity as a helpful default rather than a cure. Journals can hide names wherever they can. Reviewers can ask themselves whether they would write the same report if the name on the paper were unfamiliar. Editors can notice when a well-known author’s paper sails through on thin evidence. None of this means expertise is irrelevant or that reviewers are acting in bad faith. It means reputation is a strong signal, and a paper deserves to be judged on its own pages first.

References

[1] J. Huber, S. Inoua, R. Kerschbamer, C. König-Kersting, S. Palan, and V. L. Smith, “Nobel and novice: Author prominence affects peer review,” Proceedings of the National Academy of Sciences, vol. 119, no. 41, Art. no. e2205779119, Oct. 2022, doi: 10.1073/pnas.2205779119.

[2] A. Tomkins, M. Zhang, and W. D. Heavlin, “Reviewer bias in single- versus double-blind peer review,” Proceedings of the National Academy of Sciences, vol. 114, no. 48, pp. 12708-12713, Nov. 2017, doi: 10.1073/pnas.1707323114.