Safeguarding peer review in the era of evolving paper mills

Author: Renee Hoch, Managing Editor of the PLOS Publication Ethics Team

Trust is at the heart of the peer review system – trust that authors conduct and report research with integrity, trust that editors and reviewers provide an objective, rigorous, and fair assessment when evaluating manuscripts, and trust that publishers maintain sufficient checks and balances to uphold high scientific and ethical standards and provide researchers equitable opportunities to publish their work.

Within this system, publishers work with the research community to provide a body of published work that has been vetted by peers and to which scientists and the general public can look for reliable information pushing the envelope of knowledge.

Unfortunately, trust is sometimes broken. This week, PLOS ONE retracted 20 articles as part of a large publication ethics case involving more than a hundred published articles. The evidence in this case suggests that a limited cohort of individuals may have taken steps to manipulate – or influence the outcome of – the publication process while serving as authors, reviewers, or Academic Editors. 

Importantly, not every author, editor, or reviewer who contributed to the retracted articles was directly implicated in the concerns. This may be a case wherein upstanding researchers unknowingly became involved with a problematic service or collaborator(s) when preparing their manuscripts for publication.

Cross-publisher information sharing about specific cases and tactics can be invaluable in informing and expediting our collective efforts to safeguard peer review and ensure the reliability of the published research corpus. This post openly discusses aspects of this case in order to provide additional context for the retractions, contribute to the broader cross-publisher discussion, and raise awareness among researchers as to ‘services’ that should be avoided.   

PLOS ONE’s current case

The information below does not include all of the concerns that contributed to the retraction decisions: we must hold some information as confidential due to the nature of the concerns and because we need to protect the efficacy of our integrity checks. 

The 20 retractions PLOS ONE posted on August 3 are the first resolutions in a larger case for which most affected articles identified thus far report agricultural research. As is stated in the published retraction notices, the concerns pertain to competing interests, authorship, and peer review integrity. We identified both article-level and series-level concerns, although not every editorial concern applies to every article. 

This case opened due to concerns about an author submitting an unusually high number of submissions in a short period. Thereafter, PLOS staff editors identified the larger series on the basis of shared authors and Academic Editors. Although the majority of authors appear only once within the series, a fraction of authors and Academic Editors appear repeatedly throughout the series. PubMed and Google Scholar searches revealed that several of these individuals have published at very high volumes (e.g. >100 articles) over the past two years, exceeding what would be expected even for very active researchers.

One major issue that emerged in our investigation involved competing interests in peer review. For many articles, we found that the handling Academic Editor had recently co-published work with one or more author(s). According to the PLOS Competing Interests policy, Academic Editors should recuse themselves from reviewing the submissions in such cases. Unfortunately, this evaded PLOS ONE’s internal competing interest checks which are not currently equipped to identify co-publication histories. 

Adding to the author/editor competing interest concerns, most Academic Editors involved in the case joined the PLOS ONE Editorial Board during the period when we were receiving these submissions, were requested by the corresponding authors, and have co-published – in some cases, extensively – with other Academic Editors and/or authors who appear repeatedly across the series of affected articles. We also identified other peer review integrity concerns which we cannot discuss publicly.

Whether the issues in this case arose due to honest errors (e.g. misunderstanding of PLOS policies) or intentional systematic manipulation of the publication process, the editorial outcome is the same: we concluded that peer review was compromised and that there were editorial policy compliance issues, necessitating retraction. 

Avoiding bias in peer review by external Editorial Boards

PLOS is following up with the Academic Editors flagged in this PLOS ONE case. Looking more broadly, this case provides an example of a mechanism by which peer review involving external editorial boards can be exploited, whether by paper mills, organized networks, or individuals. We are applying our learnings from this case to inform outreach efforts and system-wide updates that can help us to mitigate this risk and help us identify similar cases prior to publication in the future. 

Importantly, while we take the threat to peer review very seriously, the concerns in this case involved only a very miniscule fraction of the PLOS ONE Editorial Board. We do not have reason to question the integrity of the many due to concerning behavior by a limited few.

PLOS ONE’s Academic Editors play a pivotal role in the publication process: they are responsible for inviting reviewers and issuing editorial decisions on submitted articles. PLOS is incredibly grateful for the service and support of our Editorial Board members, and we have a dedicated team who carefully vets Academic Editor applications, provides training during the onboarding process, and supports Academic Editors throughout their tenure with PLOS. Additionally, journal staff support Academic Editors as needed for individual manuscripts and monitor for adherence to the journal’s peer review standards, publication criteria, and policies.

Returning to the issue of trust, PLOS relies on the integrity of our Academic Editors and authors in abiding by our editorial policies and Code of Conduct for Editorial Board members. We regularly receive indications that this trust is well-placed: we often receive inquiries and editors’ recusals pertaining to potential competing interests, and we have a strong history of relationships with and peer review by our valued Editorial Board members. That said, we also receive occasional questions about competing interests. In the wake of this case we are taking steps to increase our authors’ and Academic Editors’ understanding of our Competing Interests policy and how it should be applied during peer review.

We are also looking to enhance the stringency of our internal checks. PLOS ONE’s routine competing interest checks are currently limited; to apply a check based on co-publication histories at the scale of PLOS ONE, we would need an automated tool in place, ideally one that is integrated into our submission system. We are looking toward new tools that can help enhance and broaden our competing interests checks, as well as other mechanisms  – e.g. workflow updates, audits, data-driven investigations of unusual activity – by which we can improve our detection of peer review integrity issues.

Closing remarks – a cross-publisher perspective

Systematic manipulation of the publication process appears to be on the rise, and the tactics used by bad actors continuously evolve. This presents an ever-present challenge to publishers seeking to detect such issues and ensure (or restore) the integrity of the published record. 

Whereas several large-scale cases discussed previously centered on content issues – e.g. paper mill* cases involving duplicated images, text overlap, or other erroneous or problematic content – it appears that paper mills and other author service enterprises are now inserting themselves into peer review. This PLOS ONE case has highlighted potential vulnerabilities in the peer review system that may apply across the industry to journals who rely upon external editorial boards.

In addition to addressing issues internally, we need collaborative cross-industry initiatives and advances to help us – as a collective industry – combat systematic publication manipulation more effectively and efficiently. Some such projects led by STM and COPE are now underway, and we look forward to continued progress in this area. 

*Paper mills are fee-for-service organizations that provide customers means of securing publications. These organizations (which may be formal companies, individuals, or networks of individuals) may, for example, sell fabricated manuscripts or data, authorship on articles reporting research to which the customer did not contribute, or writing and/or submission services to expedite publication of the customer’s own research by offering a guaranteed peer review outcome. 

For more discussion of the PLOS ONE case, see this RetractionWatch article. Also, research integrity is the theme of this year’s Peer Review Week. PLOS will be hosting a webinar on the topic.