On 12 September 2023, Crossref, a nonprofit that runs shared infrastructure for research publishing, announced that it had acquired the Retraction Watch database and made it a public resource. Retraction Watch is a blog that reports on retractions and keeps a database of them. Crossref’s own records held about 14,000 retractions. The Retraction Watch database held about 43,000. With some overlap, the combined set came to around 50,000.
I think this was one of the more useful things to happen to research integrity in years. A retraction is a warning label. A label that only appears on the publisher’s website, while the paper keeps circulating everywhere else without it, is not doing its job.
Three kinds of warning, in plain words
Journals have three main tools when something is wrong with a published paper, and they are easy to mix up.
A correction fixes part of a paper that is otherwise sound. The Committee on Publication Ethics (COPE) guidelines say: “If only a small part of an article reports flawed data or content, this may be best rectified by a correction.” Think of a wrong number in one table of a paper that otherwise holds up.
A retraction says the paper as a whole should not be relied on. It does not always mean fraud. COPE notes that unreliable work “may result from honest error, naïve mistakes, or research misconduct,” and that the main purpose of retraction is “to correct the literature and ensure its integrity rather than to punish the authors” (COPE). COPE advises keeping a retracted paper online with a clear notice, removing it only in extremely limited cases (COPE).
An expression of concern sits in between. It is what an editor can publish when there are serious doubts but no final answer yet. In COPE’s words, if conclusive evidence about a paper’s reliability “cannot be obtained, or will not be obtained for a significant period of time,” an editor “could consider publishing an Expression of Concern” (COPE).
Imagine a student writing a review paper who finds a striking result in a 2015 study and builds a whole section around it. She read it as a PDF a classmate shared, with no banner on it anywhere. Two years earlier, the journal had retracted it.
Each of these warnings is only as useful as the number of readers who actually see it.
Retracted papers keep getting cited
In a 2021 study in Quantitative Science Studies, Tzu-Kun Hsiao and Jodi Schneider looked at 7,813 retracted papers indexed in PubMed and how later papers cited them [1]. They found that retracted papers “continued to be cited,” and that retraction “did not change the way the retracted papers were cited” [1].
Of 13,252 places where a paper cited a retracted study after its retraction, only 722, or 5.4%, mentioned the retraction [1]. Among those 722, the retracted paper was most often cited as related work or as an example of problematic science [1]. The rest did not mention the retraction at all. The authors conclude that “the vast majority of postretraction citations in biomedicine do not document the retraction” [1].
The study covered biomedicine only, so it is not a measure of all of science. Still, the message is plain: a retraction on the journal’s website did not reach most of the people citing the work.
Why the warning gets lost
A paper travels far beyond the publisher’s page. People find it through a search engine or a database, save it to a reference manager, pass around the PDF, or copy a citation from someone else’s reference list. Each copy can miss the warning.
COPE is clear on this. Retracted articles “should be unmistakably identified as such in all online sources,” and “the retraction should appear on all online searches for the retracted publication” (COPE). In practice, coverage has been uneven. Hsiao and Schneider summarize a 2021 check of 150 retracted papers by Suelzer and colleagues: PubMed labeled every one of them as retracted, while another database, EBSCO’s CINAHL, labeled none [1].
Crossref’s announcement named the problem too. With more journals and papers than ever, tracking retractions “has become much harder for publishers and readers,” which “makes it difficult for readers and authors to know whether they are reading or citing work that has been retracted” (Crossref).
In my own work I build automated pipelines for imaging and clinical data, and the same rule applies there: a status flag that lives in only one place is a flag most downstream systems never see.
What the Crossref deal changes, and what it does not
The deal itself was modest in money. Crossref agreed to pay USD 175,000 up front and USD 120,000 a year, rising 5% each year, with an initial term of five years (Crossref).
What it buys is open, machine-readable data. Crossref said the collaboration fills “critical gaps in our coverage” and “helps the downstream services that rely on high-quality, open data about retractions” (Crossref). That is the part I care about: with open data, search tools, reference managers and databases have fewer excuses not to show the warning.
It does not fix everything. Both organizations said publishers remain “the primary stewards of the scholarly record” and should keep adding retractions to their Crossref records (Crossref). An open list helps only if the tools people actually use check it, and if authors look before they cite.
How to check a paper before you cite it
Checking takes a few minutes. Open the paper on the publisher’s site, not only from a saved PDF, and look for a retraction, correction or concern notice. Many journals show a Crossmark button, which Crossref members use to alert readers to corrections and retractions (Crossref). For biomedical papers, search PubMed, which labeled every retracted paper in the check above [1].
Your reference manager can do some of the work too. Zotero announced in 2019 that it flags retracted items using Retraction Watch data. If you try to cite one, “Zotero will warn you and confirm that you still want to cite it,” and it also warns you, when you next update citations, if a paper already cited in your document is retracted later. It worked only for items with a DOI or PubMed ID, so keep those fields filled in.
A retraction that readers never see has not really corrected anything. Journals should push the warning to every place people find papers, and anyone who cites research should check a paper’s status before citing it.
Finally, if you rely on a paper that has an expression of concern or a major correction, say so in your own text. That is the courtesy Hsiao and Schneider found missing in most citations of retracted work [1]. Science corrects itself only when the corrections travel as far as the original claims did, and each of us who cites a paper is one of the places the warning has to reach.
References
[1] T.-K. Hsiao and J. Schneider, “Continued use of retracted papers: Temporal trends in citations and (lack of) awareness of retractions shown in citation contexts in biomedicine,” Quantitative Science Studies, vol. 2, no. 4, pp. 1144-1169, Dec. 2021, doi: 10.1162/qss_a_00155.
Related Articles
