The potential butterfly effect of preregistered peer-reviewed research

Written by Lindsay Morton

Preregistration is the practice of formally registering a study design before experimentation and data collection begin. When deposited in a repository, a preregistered study design functions as a statement of intent laying claim to a certain investigatory pathway. The preregistered study design functions as an everyday tool for executing, reporting and repeating high-quality confirmatory investigations, as well as a stable methodological record capable of facilitating reproduction and reuse over the longer term. When practiced at a journal, two additional elements are added: peer review feedback, to help optimize the methods, and analytical approach; and provisional acceptance of the resulting research article when the investigation is completed, whatever the outcomes may be.

“Peer review of the registered study protocol provides an opportunity to improve the study design before starting to collect the data. That’s the most constructive time for the review to happen. And a published registered protocol is a guarantee for the authors that even null results will be published.” says Veronique Kiermer, Chief Scientific Officer at PLOS. Authors of preregistered research concur. As Jacqueline Scholl, Oxford University puts it, “”One major advantage compared to standard peer-review was that when reviewers made sensible suggestions about additional control variables to collect, we could actually implement this; rather than just thinking ‘we wished we had done that’.” 

In practical terms, neither form of preregistration outlined above is all that different from a standard research workflow. When conducting confirmatory research, investigators routinely define a hypothesis, develop methods, and choose how to collect and analyze data. These study design plans are frequently formalized as part of a grant application, and may well undergo peer review during the funding process as well.

Therefore, taking the minimal extra step of depositing a study design with a repository seems like such a small thing. A few buttons clicked, a document uploaded, and a check box or two. Initiating journal peer review earlier in the process is a similarly a minor shift; a simple reorganization of existing processes into a more efficient configuration. And yet, on the individual level, and especially in aggregate, these modest tweaks to the peer review and publication workflow have the potential to exert an outsized impact on research and scientific communication.

Privately depositing a study design with a repository helps researchers stay accountable to themselves, reducing the risk of behaviors like p-hacking, selective publication, or suppressing negative and null research outcomes. Researchers who choose to make their study designs public from the time of deposition help to avoid unnecessary repeated studies, unintentional complementary research, and scooping, freeing up their fellow researchers to pursue different lines of inquiry and improving efficiency in the field. When private repository study designs become public upon publication of a research article, their existence serves to bolster the final work, demonstrating integrity on the part of the authors, and supporting trust among readers. Finally, retaining a permanent methodological record documenting the data collection and analysis as it was actually planned and performed ensures future reproducibility as almost nothing else can.

Refocusing journal peer review on the study design phase exerts more and greater downstream changes. Peer review that focuses on evaluating significance of the research question, the methods and analytical approach before work begins, has the power to shape stronger, more rigorous and more creative research. Making an editorial decision while results are still unknown minimizes the potential impacts confirmation bias and impact bias, taking science communication back to its roots, with an emphasis on quality, rigor, and a pure intellectual curiosity. As Kiermer explains, “Preregistration and peer review of the study protocol with a journal is a way to tackle publication bias. As long as the protocol is followed, or any deviations explained, it’s a guarantee for the author that the results will be published, even if they don’t confirm their hypothesis.”

In combination, all of these factors contribute to a more complete and efficient scientific record, replete with studies exploring important hypotheses, performed to the very highest technical standards, and free from the distorting influence of impact-chasing, ego, and bias. A scientific record that is both demonstrably trustworthy, and widely trusted. And with that, there is no telling where science might go, or how quickly.