Written by Veronique Kiermer and Iain Hrynaszkiewicz Earlier this month the Open Science Monitoring Initiative shared a draft of Open Science monitoring…
A new Open Science Indicator: measuring study registration
Written by Marcel LaFlamme
With the latest release of the PLOS Open Science Indicators (OSI) results, we are introducing a new indicator for study registration—also known as preregistration. The results from this preliminary version of the indicator show that adoption is lower than for other indicators but is growing, albeit slowly. Over time, more researchers may be discovering how registering and publicly sharing a study design before results are known can enhance trust in their work.
This release also provides a first look at Open Science practices for articles published in 2024, bringing the results for data sharing, code sharing, and preprint posting up to date through the first quarter of this year. The complete dataset, which extends back to 2018 and now comprises 135,214 articles, is as always freely available to access and reuse.
Developing the new indicator
Study registration has been identified as a priority Open Science practice to monitor in biomedical research. It is also an indicator of interest for the pilots being organized by the UK Reproducibility Network, in which PLOS and our partner DataSeer are participating. For the purposes of OSI, we defined study registration as “the plan for a research study, including research questions/ hypotheses, details about the research design, and/ or plans for data analysis, which has been made available for public sharing in order to ensure unbiased reporting and support the differentiation of planned and unplanned research directions.”
From there, we worked to develop an approach to measuring study registration in the published scientific literature. We initially focused on detecting references to registries that record structured information about a study’s design, including clinical trial registries, systematic review registries, animal study registries, and other general purpose registries like the Open Science Framework. We then matched persistent identifiers and other links with the corresponding registries.
To our knowledge, this approach—covering more than 30 registries—is the most comprehensive one that’s been developed and implemented at scale. But, because it relies on a predefined allowlist of registries (see the study registration indicator’s methods statement for more information), it’s possible that there are registries we missed. If you have any feedback on the approach we’ve taken, please let us know at mlaflamme [at] staging.plos.org.
How common is study registration?
Over the past five years, study registration rates at PLOS have risen slowly but steadily, from 5% of articles in 2018 to 7% in 2023. The same overall trendline can be observed in the comparator corpus, with rates rising from 6% in 2018 to 7% in 2023.
Figure 1. Study registration rates for PLOS and comparator content by publication quarter, 2018-2023.
The most commonly used registry by PLOS authors is Clinicaltrials.gov, followed by the systematic review registry PROSPERO. But the prevalence of registrations using the Open Science Framework and other registries, covering a wide variety of study designs, can also be observed for the first time with this new indicator. The data being released are currently not optimized for segmenting by discipline, although a previous version of the OSI dataset does include fields of research for most of the relevant articles and we plan to share an analysis of the study registration data by field later this year. Other features we are considering for this indicator include detecting registrations in the text of supplementary files and capturing the dates when registrations were recorded and shared. But what else would make the indicator more useful? Again, we’d be grateful for your input.
Other results and refinements
The main data files in the latest OSI dataset include all 112,229 PLOS research articles published between 1 January 2018 and 31 March 2024, as well as a smaller comparator corpus. For PLOS articles the latest results show:
- 31% of all research articles shared research date in a repository in Q1 2024 (no change from Q4 2023)
- 17% of all research articles shared code in Q1 2024 (up 1% from Q4 2023)
- 22% of all research articles were associated with a preprint in Q1 2024 (down 2% from Q4 2023)
Figure 2. Data sharing in a repository, code sharing, and preprint posting for PLOS and comparator content by publication quarter, 2018-Q1 2024.
The rate of preprint adoption at comparators has been higher than at PLOS since 2023, driven by publishers with in-house preprint servers, but this rate also dropped in comparators in Q1 2024, suggesting a wider trend. Meanwhile, improvements have been applied to the preprint indicator and applied retrospectively to the entire corpus. A minor source of false positives has been removed and, more significantly, preprints posted after the associated research article was published (also known as “postprints”) have been eliminated. This change disproportionately affects older articles, which have had longer to accumulate a postprint. See the updated methods statement in the main documentation for more information.
We’ve also updated our comparator article matching approach. For this latest version, the Medical Subject Headings (MeSH) terms for the new quarter of data have been updated based on the distribution of MeSH terms for PLOS articles published between 1 January 2022 and 31 December 2023. We expect to update the MeSH term distribution yearly to ensure that the comparator sample remains comparable to the changing composition of PLOS articles.