On September 12, 2023, Crossref made an announcement. Crossref is a nonprofit that runs shared systems for research publishing. It said it had acquired the Retraction Watch database and made it a public resource. Retraction Watch is a blog that reports on retractions and keeps a database of them.
Crossref’s own records held about 14,000 retractions. The Retraction Watch database held about 43,000. Some papers were in both. Together, the combined set came to around 50,000.
I think opening up this database was one of the most useful steps for research integrity in years. A retraction is meant to warn readers that a paper has been withdrawn. If that warning only appears on the publisher’s website, and copies of the paper keep spreading elsewhere without it, many readers will never see it.
What are the three kinds of warning?
Journals have three main tools when something is wrong with a published paper. They are easy to mix up.
A correction fixes part of a paper that is otherwise sound. An example is a wrong number in one table, when the rest of the paper is correct. The Committee on Publication Ethics (COPE) guidelines explain it. “If only a small part of an article reports flawed data or content, this may be best rectified by a correction.”
A retraction says the paper as a whole should not be relied on. It does not always mean fraud. COPE notes that unreliable work “may result from honest error, naïve mistakes, or research misconduct” (COPE). It says the main purpose of retraction is “to correct the literature and ensure its integrity rather than to punish the authors” (COPE).
COPE advises keeping a retracted paper online with a clear notice (COPE). It should be removed only in extremely limited cases (COPE).
An expression of concern is a middle step between the two. An editor can publish one when there are serious doubts but no final answer yet. Sometimes firm evidence about a paper’s reliability is missing. COPE describes the case where it “cannot be obtained, or will not be obtained for a significant period of time” (COPE). Then, in COPE’s words, an editor “could consider publishing an Expression of Concern” (COPE).
Imagine a student writing a review paper who finds a striking result in a 2015 study and builds a whole section around it. She read it as a PDF a classmate shared, with no banner on it anywhere. Two years earlier, the journal had retracted it.
Each of these warnings helps only the readers who actually see it.
Do people keep citing retracted papers?
In 2021, Tzu-Kun Hsiao and Jodi Schneider published a study in Quantitative Science Studies [1]. They looked at 7,813 retracted papers listed in PubMed, a large database of biomedical research. They checked how later papers cited them. They found that retracted papers “continued to be cited,” and that retraction “did not change the way the retracted papers were cited”.
The researchers found 13,252 places where a paper cited a retracted study after its retraction. Only 722 of them, or 5.4%, mentioned the retraction. That is about 1 in 18. Among those 722, the retracted paper was most often cited as related work, or as an example of problematic science.
The rest did not mention the retraction at all. The authors conclude that “the vast majority of postretraction citations in biomedicine do not document the retraction”.
The study covered biomedicine only, so it does not measure all of science. Still, it shows that a retraction notice on the journal’s website did not reach most of the people citing the work.
Why does the warning get lost?
People read a paper in many places besides the publisher’s website. People find it through a search engine or a database. They save it to a reference manager, pass around the PDF, or copy a citation from someone else’s reference list. Any of these copies can be missing the warning.
The COPE guidelines say the warning should appear everywhere. Retracted articles “should be unmistakably identified as such in all online sources” (COPE). Also, “the retraction should appear on all online searches for the retracted publication” (COPE).
In practice, some databases show the warning and others do not. Hsiao and Schneider summarize a 2021 check by Suelzer and colleagues of 150 retracted papers [1]. PubMed labeled every one of them as retracted. Another database, EBSCO’s CINAHL, labeled none.
Crossref’s announcement described the same problem. There are more journals and papers than ever. So tracking retractions “has become much harder for publishers and readers” (Crossref). This “makes it difficult for readers and authors to know whether they are reading or citing work that has been retracted” (Crossref).
People who build computer systems know a similar rule. If a warning is stored in only one place, most other systems that copy the data never see it.
What does the Crossref deal change?
The deal did not cost much money. Crossref agreed to pay USD 175,000 up front (Crossref). It also agreed to pay USD 120,000 a year, rising 5% each year, for a first term of five years (Crossref).
What it buys is open data that computers can read. Crossref said the deal fills “critical gaps in our coverage” (Crossref). It also said it “helps the downstream services that rely on high-quality, open data about retractions” (Crossref). In my view, this open data is the most important part. With open data, search tools, reference managers and databases have fewer excuses not to show the warning.
It does not fix everything. Both groups said publishers remain “the primary stewards of the scholarly record” (Crossref). They said publishers should keep adding retractions to their Crossref records (Crossref). An open list helps only if the tools people actually use check it. It also helps only if authors look before they cite.
How can you check a paper before you cite it?
Checking takes a few minutes. Open the paper on the publisher’s site, not only from a saved PDF. Look for a retraction, correction or concern notice. Many journals show a Crossmark button. Publishers that belong to Crossref use it to alert readers to corrections and retractions (Crossref). For biomedical papers, search PubMed, which labeled every retracted paper in the check above [1].
Your reference manager can do some of the work too. Zotero announced in 2019 that it flags retracted items using Retraction Watch data. If you try to cite a retracted item, “Zotero will warn you and confirm that you still want to cite it,” the Zotero team wrote. A paper you already cited might also be retracted later. Zotero warns you about that when you next update citations. It worked only for items with a DOI (a permanent ID number for a paper) or a PubMed ID. So keep those fields filled in.
In my view, a retraction that readers never see does not correct the record. Journals should send the warning to every place people find papers. Anyone who cites research should also check a paper’s status before citing it.
Finally, you may rely on a paper that has an expression of concern or a major correction. If so, say so in your own text. Hsiao and Schneider found that most citations of retracted work gave readers no such note. Science corrects itself only when the corrections travel as far as the original claims did. Everyone who cites a paper needs to see the warning and pass it on.
References
[1] T.-K. Hsiao and J. Schneider, “Continued use of retracted papers: Temporal trends in citations and (lack of) awareness of retractions shown in citation contexts in biomedicine,” Quantitative Science Studies, vol. 2, no. 4, pp. 1144-1169, Dec. 2021, doi: 10.1162/qss_a_00155.
Related Articles
