I am excited to share with our community that we have completed a recent series of appointments to PLOS’ Board of Directors. 2023…
Transparency and Trust: Peer Review Week 2020
This post is written by PLOS’ Chief Scientific Officer Veronique Kiermer
Peer Review Week is a time to celebrate the outstanding work of reviewers across the scholarly ecosystem. But it also serves as a call to action to examine the systems we have in place and opportunities to improve.
This peer review week, the theme couldn’t feel more timely. Trust in science, and peer review more specifically has taken the spotlight as doctors, researchers, and policymakers around the world collaborate to rapidly expand and communicate emerging knowledge about COVID-19.
At the same time, the advent of preprints in biomedical sciences has created new challenges to evaluate findings outside the traditional framework of journal peer review, and high profile journal retractions have contributed to raise doubts about the effectiveness of pre-publication peer review. With the flood of information pouring in, public readers and researchers alike face the same question: how do I identify information I can trust?
Does Peer Review Work?
Peer review has long been a staple of the credibility of scientific findings for researchers and for the public. The process embodies the norm of organized skepticism that helps generate trust in scientific findings—by testing the rigor of thinking and experimentation, and by providing authors with expert feedback to build on. But importantly, peer review is more than the traditional view of pre-publication peer review conducted behind closed doors. Peer review does not stop with publication—in many ways, it only starts there.
Recently, in the context of the COVID-19 pandemic, we’ve seen researchers rapidly come together after the publication of high-stake results and point out serious methodological flaws which have led to retractions of articles and withdrawals of preprints. Social media has been an unconventional, yet instrumental tool in coordinated reactions. Some have argued this is a failure of peer review, but we believe it’s actually a celebration of peer review in different forms.
A broader view of peer review
While post-publication peer review remains rare, preprints offer new opportunities to leverage the community in order to address the challenges and limitations of traditional peer review. For example, reviewers and editors commonly acknowledge that it is increasingly difficult to cover all the bases of expertise required to properly analyze articles that have become much more multidisciplinary. Preprint commenting while the article is under review is one possible solution we’ve encouraged to address this gap, by opening up the opportunity for any researcher, anywhere in the world to volunteer their expertise, for consideration by the editor. While commenting on preprints is still rare, it could allow journals to channel within the traditional process the energy that the community already puts in scrutinizing new findings within their specialty before they are formally published in journals.
Preprints and preprint commenting can also help with another challenge of journal review: speed. PLOS journals offer authors the option to post preprints on their behalf at submission—after minimal checks for any content with potential to cause harm—so their findings can be disseminated while the journal curation process takes place. We also participate in initiatives like Review Commons to provide journal-agnostic reviews that can accelerate the peer review process at the author’s destination journal and alleviate the strain of repeated reviews on the peer reviewer community.
Beyond these practical issues, we must consider systemic challenges and in particular the documented biases in peer review. At PLOS we acknowledge the need to improve the diversity of our reviewer pool to help mitigate such biases and we are committed to this work.
Transparency, transparency, transparency
An important prerequisite for all peer review mechanisms to work as effectively as they can is transparency.
Transparency facilitates trust. Working to make every step of the research and publication process as Open as possible gives journals, peer reviewers, and the scholarly community more broadly, the means for scrutiny, and a clear understanding of limitations. At the same time, it gives readers deeper insight into the scientific process and the various checkpoints that uphold it.
From an author’s perspective, increasing transparency means making all relevant data, code, methodologies, and reagents available so that the scholarly community—starting with reviewers—can validate and reproduce the findings independently. A recent survey by the Center for Open Science indicated that researchers judging preprint credibility were influenced by cues related to open data, code and other openness measures, as well as independent verification of authors claims.
Publishing peer review reports as well is important as it makes evident what aspects of the article were (and were not) thoroughly scrutinized pre-publication. All PLOS journals now encourage their authors to publish their peer review history alongside the article, and the uptake is promising.
Peer review needs better recognition
Transparency in peer review is also crucial to recognizing peer review as a first-rate academic output. It is time for this activity, essential to the scientific process, to be considered a scientific output in its own right and to take its due place in the context of research and researchers assessment. With published peer review reports, individual reviewers can choose to take credit by revealing their identity. But recognizing that there are circumstances that call for anonymity, even unnamed reviewers should have this opportunity, and tools like ORCID and Publons allow them to demonstrate their contributions as part of their academic profile.
There’s always more we can do to provide the motivation and support for reviewers’ work. Peer review transparency and other visual signs of recognition are a way to start helping researchers leverage their peer review expertise to advance their careers.
Earning trust in peer review
Despite its challenges, the traditional journal peer review process offers important opportunities to ensure fairness in peer review. And journals have a collective responsibility to make use of these opportunities. First and foremost, we can ensure that every paper reporting a rigorously designed and executed study is peer reviewed—not only the high-stakes findings and the work of well known authors. We can also take proactive measures to limit bias, such as increasing the diversity of reviewer pools, and managing competing interests. We can foster consistent review criteria, and supplement the volunteer peer review reports with additional scrutiny for elements like statistics or ethical research practices.
In sum, I believe there are complementary benefits of journal-organized peer review and the process that can take place more organically in the research community post-publication—whether the publication is in a journal or on a preprint server. With more transparency in the system, we can take advantage of all mechanisms of peer review to decide what to trust, and importantly peer review itself can deserve our trust.
I am very grateful to the reviewers who helped improve the article and eliminate the shortcomings we did not notice.
Reviewers of Springer Nature, Manuscript Central and Wiley are excellent critics. They have brought submerged research findings for readers and helped in adding knowledge of medical science to people. They helped researchers in documenting their results in proper way.
I thank them all from my heart.
Peer review and transparency of information has raised the level of scientific work. The commitment of researchers through the review processes is of great value to science, giving greater credibility to the published results. We hope you continue on this path.
In the Medical Sciences, Peer Review has warped into a tool that seeks to promote and sustain faulty ideas far beyond their demise.
It seeks to maintain an established status quo, and eschews innovation that challenges the status quo. It serves to sustain an established power structure in the administrative aspects of Medical Sciences, and has far strayed from the honorific principles It is designed to uphold.
Cynical views of its process are duly warranted.
This article is so inspiring!
Thank you so much
I can confirm substantial learning progress in revising my recent paper I submitted to PLOS. With former papers in other journals I had also less enjoyable experiences.
We are highly thankful to the reviewers for their helpful comments which improved the quality of the research paper.
The journal is well built trust on researchers including me. Thus, i appreciated your rapid response and commitments in the review process.
Nice Reading Material for a senior citizen who just started to learn about the Scientific way of reviewing of commenting post printed materials.
It’s now I really appreciate peer review in its full context.
I much appreciate the reviewers and academic editor to support our work to born in the world.
Great effort. Peer review needs to be recognized at larger platforms.
Reviewing a scientific paper is a task that requires the expertise of not only in the field of the papers’ theme but in being an honest and fair critic where the comments raised are intended to add to the scientific value of the paper and greatly helps the authors. A bug salutation to such reviewers.
This comment written by Dr. Veronique Kiermer of PLOS’ Chief Scientific Officer is very siginificant for the present science and science field confused by covid-19. We here have understood that publishing peer review reports and building new community by preprints will be likely to affect scientists all. By your comment, we also remember the article of why most published research findings are false (J.P.A. Ioammidis, PLos Medicine, 2(8): e124.).
I think that the real turning point will come when the reviewers’ work is paid for by the journals. Only university staff who have a good and permanent salary can afford to invest a lot of time in reviewing articles. The work of a reviewer is a painstaking job that often requires further updating; moreover, it is mostly anonymous and thus it does not receive sufficient acknowledgements. It is not possible to think that he is not paid. That is why many journals, recently, have a hard time to find reviewers.
I think that generally the peer review improve the paper. Some times they are not usefull because they donot know deeply the topic. In general make me feel sure.
For me, reviewers are very important elements for the dissemination of research and they should be proud of their silent functions. My thanks for all that reviewed my papers recently published. Achiléa Bittencourt
The Journal is very nice. I am happy on it since it assists emerging countries (Africans)
Peer review and preprint commenting is an essential process for the credibility of scientific findings for researchers and for the public. Since preprint comment is not common much efforts need to be in place to make it as a scientific culture.