Bringing Peer Review Out of the Shadows

An interesting collection of reader comments has accumulated in response to the PLoS Biology article by Hauser and Fehr, entitled “An incentive solution to the peer review problem”, published earlier this year.

Hauser and Fehr propose a system for holding late reviewers to account by penalizing them when it’s their turn to be an author. A slow reviewer’s paper would be “held in editorial limbo” for a length of time that reflects their own tardiness as a reviewer. The short article was intended to provoke a discussion about how to improve peer review – an opening card as Hauser and Fehr put it.

So far, 16 responses have been added from readers, and the general view seems to be that incentives would be more effective than the punishment that Hauser and Fehr propose. As for an incentive, quite a number of respondents favour a system whereby reviewers are paid for their efforts – although not to a level that would fully reimburse the time spent.

It’s interesting to consider this payment idea in light of the publication fee model that many open access journals use. The essence of this model is that the publication fees cover the costs of the publishing process. So if we were to add to those costs by paying reviewers, then the publication fee would need to be raised as well. Given that the author and reviewer populations overlap to a large extent, this approach would end up shuffling money around within the community of researchers—not a particularly attractive solution to the problem.

But maybe money isn’t the best incentive anyway. And in any case, the challenges facing peer review aren’t just about the speed of the process. Several of the respondents to the discussion emphasize the importance of the quality of the reviews; it’s worth waiting a while longer if the end result is a more considered set of comments on the article. To focus any incentive system on speed alone won’t necessarily improve the quality of peer review one jot.

Maybe a different way to think about the question is to consider who should be the judge of whether or not a reviewer does a good job. As an editor, I’ve been privileged to be party to some incredibly thoughtful and constructive discussions between authors and reviewers. It’s not always that way of course, but when it works, it’s fantastic. I’ve often felt that it’s a pity that these exchanges haven’t been shared more broadly, and as pointed out by others in the discussion, there are many who feel that greater transparency in peer review is the way to go – pre- and post-publication. We are seeing more and more experiments in this arena – PLoS ONE being the obvious example close to home, and Atmospheric Chemistry and Physics being one of the trailblazers – and one can certainly envisage incentives being developed in these more transparent systems. Researchers who add their thoughts to published articles can be rated by other members of their community on the basis of the quality and usefulness of their comments. Their comments can be cited in their own right, ultimately adding to their scientific reputation and credentials – a powerful incentive indeed.

The formula for more transparent peer review might not be perfect yet, but there is great potential and further experimentation is a must. Ultimately, improving the peer review process will take the same kind of thoughtful and constructive discussions that help researchers identify the extra step that will maximize the significance of their results. We invite you to join in that discussion.