Europe PMC started indexing preprint articles in 2018 alongside journal peer-reviewed articles. During the COVID-19 pandemic preprints became increasingly popular as they offered immediate access to resarch findings and data. But it was left up to the reader to make a judgement about the robustness of the science as no peer reviews were available. Several different preprint platforms started to enable readers to evaluate and review preprints. Europe PMC had the unique opportunity to display aggregated peer reviews on preprints.
The challenge
Before adding open peer reviews for preprints in Europe PMC, the team needed to understand peoples’ perceptions towards preprint reviews and pain points with reading and evaluating preprints. Traditional journal peer review assesses the validity and quality of the research. Journal peer reviews are usually only visible to the article authors and journal editor and are conducted by invited reviewers who are experts in the research area.
In contrast, preprint peer reviews are open to everyone and can vary in quality and rigour. We hypothesised that readers could potentially be confused about the validity of preprint peer reviews and the review process. And that authors could feel uncomfortable about reviews being publicly available.
Some preprints go through several revisions and the versions are linked together on Europe PMC. Any of the preprint versions could have a peer review of it.
This work built on collaboration with a wider community of preprint review platforms, peer review aggregators and preprint servers, following a workshop hosted by ASAPBio (a group who advocate for preprints) in 2023: Supporting interoperability of preprint peer review metadata. I helped ASAPBio to prepare for and facilitate this workshop. One of the outputs was a community roadmap for preprint peer review interopability.
My role as User Researcher
To test our hypotheses I designed a simple prototype which included a search results page, preprint page and peer review page. I conducted remote usability tests with 9 participants using the prototype to get feedback. The goals of the research were to understand:
- Users’ experiences of preprints and open peer review
- Users’ attitudes to seeing peer review events
- Where do users expect to find open peer review information on article pages?
- What peer review information is useful/valuable?
- How should Europe PMC display peer review information?
- How should Europe PMC handle reviews in relation to preprint versions?
- Is providing a link to read peer review on external sites sufficient?
- Do users want to see / filter preprints with peer reviews when scanning search results?
I wrote up the findings of this research and shared it openly in this peer review user research report.
I worked closely with our UX Designer to iterate my intial designs, in particular how peer reviews were displayed for different versions of the preprint.
My role as Product Manager
I worked closely with our Data Scientist, Development team and external collaborators to define and review the technical specification. It was important to ensure the right information could be presented on the Europe PMC website. A process was created to pull information about each review from preprint review aggregators Sciety and EMBO’s Early Evidence Base. A piece of open source software called docmap-parser was developed, which converts a DocMap file into the XML format used by the Europe PMC database.
Outcomes
The new functionality to display preprint peer reviews on Europe PMC was released in late 2023. As of April 2025 there are now 16,614 preprints with peer reviews in Europe PMC.
The Europe PMC team wrote a paper which was published after I left the organisation: Enabling preprint discovery, evaluation, and analysis with Europe PMC.
References
Rosonovski, S. Discovering Reviewed Preprints. 2024, Jan.
Levchenko M, Parkin M, McEntyre J, Harrison M. Enabling preprint discovery, evaluation, and analysis with Europe PMC. Plos one. 2024 ;19(9):e0303005. DOI: 10.1371/journal.pone.0303005. PMID: 39325770; PMCID: PMC11426508.