Preprints are articles that have not passed formal peer review or been published in a journal. Europe PMC started indexing preprint articles in 2018, alongside peer-reviewed journal articles.
During the COVID-19 pandemic preprints became increasingly popular as they offered immediate access to research findings and data. But it was left up to the reader to make a judgement about the robustness of the science, as no peer reviews were available. Several different preprint platforms started to enable readers to evaluate and review preprints openly. Europe PMC had the unique opportunity to display aggregated peer reviews on preprints.
The challenge
Before adding open peer reviews for preprints in Europe PMC, the team needed to understand peoples’ perceptions towards preprint reviews and their pain points with reading and evaluating preprints.
Traditional journal peer review assesses the validity and quality of the research. Journal peer reviews are usually only visible to the article authors and journal editor and are conducted by invited reviewers who are experts in the research area.
In contrast, preprint peer reviews are open and can vary in quality and rigour. During the pandemic preprints were being used by journalists to report on the latest research findings. Our assumptions were that:
- Readers might be confused about the validity of preprints with peer reviews.
- Readers may not understand the preprint review process and how it differs from journal peer review.
- Readers may not trust preprint peer reviews.
- Authors could feel uncomfortable about reviews being publicly available.
A complexity of preprint reviews is that some preprints go through several revisions and the versions are linked together on Europe PMC. Any of the preprint versions could be reviewed.
Background
This work built on our collaboration with a wider community of preprint review platforms, peer review aggregators and preprint servers, following a workshop hosted by ASAPBio (a group who advocate for preprints) in 2023: Supporting interoperability of preprint peer review metadata. I worked with ASAPBio to prepare for and facilitate this workshop. One of the outputs was a community roadmap for preprint peer review interopability.
Discovery
To test our hypotheses I designed a simple prototype which included a search results page, preprint page and peer review page. I conducted remote usability tests with 9 participants using the prototype to get feedback.
I shared my research findings with our partners at eLife and EMBO and the wider preprint community in this peer review user research report. Users found the timeline view helpful, but they expected to see the reviews before the author response, rather than seeing the most recent item first. I identified several indicators that helped build trust, for example showing preprint server logos, providing more detail about expert sources and grouping reviews from the same platform.
I worked closely with our UX Designer to iterate my intial designs, in particular how peer reviews were displayed for different versions of the preprint.
Technical specification
I worked closely with our Data Scientist, Development team and external partners to define and review the technical specification. It was important to ensure the right information could be presented on the Europe PMC website.
A process was created to pull information about each review from preprint review aggregators Sciety and EMBO’s Early Evidence Base. A piece of open source software called docmap-parser was developed, which converts a DocMap file into the XML format used by the Europe PMC database.
I left EMBL-EBI in October 2023. EMBL-EBI is an intergovernmental organisation and has a maximum 9 year contract rule. My 9 year contract sadly came to an end, therefore I did not manage the full delivery of the preprint peer review project. The new functionality to display preprint peer reviews on Europe PMC was released in late 2023.
Outcomes
As of November 2025 there are now 17,975 preprints with peer reviews in Europe PMC.
The Europe PMC team wrote a paper which was published after I left the organisation: Enabling preprint discovery, evaluation, and analysis with Europe PMC.
References
Rosonovski, S. Discovering Reviewed Preprints. 2024, Jan.
Levchenko M, Parkin M, McEntyre J, Harrison M. Enabling preprint discovery, evaluation, and analysis with Europe PMC. Plos one. 2024 ;19(9):e0303005. DOI: 10.1371/journal.pone.0303005. PMID: 39325770; PMCID: PMC11426508.