Category: Work

Europe PMC started indexing preprint articles in 2018 alongside journal peer-reviewed articles. During the COVID-19 pandemic preprints became increasingly popular as they offered immediate access to resarch findings and data. But it was left up to the reader to make a judgement about the robustness of the science as no peer reviews were available. Several different preprint platforms started to enable readers to evaluate and review preprints. Europe PMC had the unique opportunity to display aggregated peer reviews on preprints.

The challenge

Before adding open peer reviews for preprints in Europe PMC, the team needed to understand peoples’ perceptions towards preprint reviews and pain points with reading and evaluating preprints. Traditional journal peer review assesses the validity and quality of the research. Journal peer reviews are usually only visible to the article authors and journal editor and are conducted by invited reviewers who are experts in the research area.

In contrast, preprint peer reviews are open to everyone and can vary in quality and rigour. We hypothesised that readers could potentially be confused about the validity of preprint peer reviews and the review process. And that authors could feel uncomfortable about reviews being publicly available.

Some preprints go through several revisions and the versions are linked together on Europe PMC. Any of the preprint versions could have a peer review of it.

This work built on collaboration with a wider community of preprint review platforms, peer review aggregators and preprint servers, following a workshop hosted by ASAPBio (a group who advocate for preprints) in 2023: Supporting interoperability of preprint peer review metadata. I helped ASAPBio to prepare for and facilitate this workshop. One of the outputs was a community roadmap for preprint peer review interopability.

My role as User Researcher

To test our hypotheses I designed a simple prototype which included a search results page, preprint page and peer review page. I conducted remote usability tests with 9 participants using the prototype to get feedback. The goals of the research were to understand:

• Users’ experiences of preprints and open peer review
• Users’ attitudes to seeing peer review events
• Where do users expect to find open peer review information on article pages?
• What peer review information is useful/valuable?
• How should Europe PMC display peer review information? 
• How should Europe PMC handle reviews in relation to preprint versions? 
• Is providing a link to read peer review on external sites sufficient? 
• Do users want to see / filter preprints with peer reviews when scanning search results?

I wrote up the findings of this research and shared it openly in this peer review user research report.

I worked closely with our UX Designer to iterate my intial designs, in particular how peer reviews were displayed for different versions of the preprint.

My role as Product Manager

I worked closely with our Data Scientist, Development team and external collaborators to define and review the technical specification. It was important to ensure the right information could be presented on the Europe PMC website. A process was created to pull information about each review from preprint review aggregators Sciety and EMBO’s Early Evidence Base. A piece of open source software called docmap-parser was developed, which converts a DocMap file into the XML format used by the Europe PMC database.

Outcomes

The new functionality to display preprint peer reviews on Europe PMC was released in late 2023. As of April 2025 there are now 16,614 preprints with peer reviews in Europe PMC.

The Europe PMC team wrote a paper which was published after I left the organisation: Enabling preprint discovery, evaluation, and analysis with Europe PMC.

References

Rosonovski, S. Discovering Reviewed Preprints. 2024, Jan.

Levchenko M, Parkin M, McEntyre J, Harrison M. Enabling preprint discovery, evaluation, and analysis with Europe PMC. Plos one. 2024 ;19(9):e0303005. DOI: 10.1371/journal.pone.0303005. PMID: 39325770; PMCID: PMC11426508.

Europe PMC is an open, life sciences literature discovery tool that supports the open access policies of 37 funders. The primary users are life sciences researchers (academic and industry), clinicians and biocurators.

The challenge

When I joined the team they had limited knowledge of user-centred design, no user research insights or behavioural analytics and lots of assumptions.

Articles in Europe PMC are linked to research outputs such as supporting data, code, protocols and other resources including peer review reports. Biological terms and concepts such as gene-disease relationships and biological samples are text mined from the full text and highlighted on the article text. 

The aim of the redesign was to better support users’ literature search goals and tasks and surface linked research outputs and value-added resources that are the unique selling point of Europe PMC.

My role as User Researcher

To understand the users’ goals, tasks and behaviours when literature searching I planned and conducted a generative research study in 2017 which included:

• A diary study with 13 participants: participants were asked to record their literature searches in a Google Docs template during one working week. Working with the Junior UX Designer, we interviewed the participants before their journalling activity. We then analysed their diary study responses and interviewed participants again at the end of the week. I collated all the search queries in one spreadsheet and categorised the types of search.
• Usability testing: participants’ were asked about their literature searching behaviour and recent literature searches. They were then asked to complete tasks related to recent literature searches using the search tool they use most frequently, and then to repeat the tasks on Europe PMC.
• Competitor analysis: to learn what features were available in other literature search tools.

85 searches were analysed in the diary study. Five main search types were identified including: exploring a new topic; finding new articles on a familiar topic, finding evidence within articles, finding a specific article and finding information about a methodology.

I added all the insights to a Trello board with participant quotes by theme, for quick reference and sharing with the team.

I created a search experience map and a research report to share with the team and stakeholders.

I created a task model to break down the researcher goal: finding evidence to inform a research question. This helped to analyse the thought processes and behaviours of a researcher doing a literature search. Which in turn informed our search functionality and results page design.

I also created personas and search scenarios to help the team understand different motivations and pain points of researchers and biocurators, when searching the literature.

The research insights led to a complete redesign of the article and search pages. I collaborated with the UX Designer to develop several different designs and we tested them iteratively with users.

Some of the design improvements included:

• Improved search results design including labelling of different article types and enhanced search filter functionality
• Article page redesign including:
  • Figure previews under the abstract
  • A data section containing links to supplemental and cited or curated data
  • Re-organising links to useful resources elsewhere under sections that made more sense to users including reviews, protocols,
  • A citations and impact section including data citations

My role as Product Manager

Once the designs had been through a few rounds of usability testing, I led an Agile team of 5 full stack and front-end developers, 1 QA / DevOps and the UX Designer to deliver the project.

I created and managed the product backlog of user stories in JIRA and facilitated sprint planning meetings, sprint retrospectives and release planning meetings. I worked with our QA to define our manual testing approach.

The project provided an opportunity to modernise the front-end tech stack which was previously based on the Wicket framework. After some technical investigation and evaluation we decided to use Vue.JS and a design system created in Storybook. The front-end was re-built in stages. We also used Rendertron to render all the site content without CSS for search engines.

I worked collaboratively with the new UX Designer and front-end developer on developing the page designs and interactions further, creating a design system and iterative testing with users.

Due to the scale of the project I planned a phased release schedule. We started with a beta release so that we could get early feedback on the new design and features before the general release.

Anonymous usage data from Europe PMC (e.g. user interactions with links, buttons or form elements), were tracked using Piwik. I defined several custom events and worked with the front-end developer to implement them. I analysed quantitative usage data and defined metrics to measure the impact of the redesign.

Outcomes

Satisfaction ratings for regular users increased by 11% from 66.8% in 2017 to 77.3% in 2021. Engagement by repeat users increased and the number of unique users per year tripled from 2017 to over 30 million. 

Evidence from the user research shifted conversations with stakeholders and changed priorities on the roadmap.

“There is huge praise across the Scientific Advisory Board for the new features of the user interface and the approach to systematically and continuously collect user feedback that informs product development.”

Europe PMC Scientific Advisory Board 2020 report

Learnings

The biggest learning curve from this project was how to manage the release of a major site redesign. We tested the functionality of the site thoroughly, using bug bashes as well as manual and some automated testing by our team QA. I’ve written about bug bashes here.

I planned four different phases of release. As part of this schedule, a beta version of the site was released and linked from the site header so that we could get early feedback on the new design and functionality. The launch date of the new site was publicised on Twitter. We felt confident about our plans.

One of the underlying technologies we were using was Rendertron, which renders the entire site content for search engines on the fly, without CSS. It worked well on the staging environment. However when we moved the site to the live environment there was a performance issue. We had page loading speed KPIs to meet, so we couldn’t go publicly live until we had fixed the issue and improved page loading speed.

It took the technical team around a week to resolve the issuess at launch. I ran a retrospective with the team to reflect on what we could have done to avoid the issues. A key lesson learned was that better load testing with more realistic session data would have helped. We took that learning forward for future projects.

A new UX Designer joined the project after the initial user research had been completed. It was challenging for her to get up to speed with the findings of the research. Her understanding improved over time as she was involved in more of the usability tests.

References

Ferguson C, Araújo D, Faulk L, et al. Europe PMC in 2020. Nucleic Acids Research. 2021 Jan;49(D1):D1507-D1514. DOI: 10.1093/nar/gkaa994. PMID: 33180112; PMCID: PMC7778976.

PMC, Europe; Ide-smith, Michele (2017). Europe PMC Literature Search User Research Report. figshare. Journal contribution. https://doi.org/10.6084/m9.figshare.4789744.v1

PMC, Europe; Ide-smith, Michele (2017). Europe PMC Search Experience Map. figshare. Poster. https://doi.org/10.6084/m9.figshare.4789738.v1

Europe PMC Team. The new Europe PMC is here. 2019 Dec.

Europe PMC provides access to millions of life sciences research articles. It can be challenging to find articles by a particular author or co-author, particularly if they have a common name. To resolve this issue, Europe PMC integrates with ORCID, the registry of unique, persistent identifiers for academic researchers. Authors can maintain their own ORCID record by ‘claiming’ articles they have authored through integrated services such as Europe PMC.

To promote the value of ORCID iDs for researchers and increase uptake of the ORCID features in Europe PMC, my manager proposed creating author profile pages for authors.

The challenge

For this project the biggest challenges were:

• how to make author profiles discoverable from the search results in Europe PMC
• what information to include in the profiles
• how to communicate to authors how the profiles had been created and how to manage which publications were listed on their profiles

My role as User Researcher and Designer

Based on my understanding of what information was available using the Europe PMC search by ORCID id, I created some prototype designs for the profiles. These started as paper prototypes and developed into higher fidelity visual designs as below.

I carried out remote and in person moderated usability testing with life sciences researchers using paper and digital prototypes. The testing provided several useful insights, for example:

• Confusion with the terminology used to describe open access and free full text articles in Europe PMC.
• How participants expected to be able to interact with the charts and filter or download information.
• Participants assumed it would be a public profile page and felt it would be more useful if they could look at other researcher profiles, similar to Google Scholar.
• Expectations about what types of publication should be included, and how citations data should be presented.

As discoverability of the profiles was important I sketched out a flow for how profiles could be accessed and linked from different places.

I then developed this concept into higher fidelity mock-ups with annotations for the developers.

Outcomes

Author profiles were released in 2016. An example author profile.

The profiles of highly cited authors are linked from the search results page.

Learnings

The biggest learning from this project was that researchers were not always comfortable with an author profile being created automatically for them. They wanted to edit the profile and curate their list of publications or hide the profiel completely. We assumed at the start that authors would want control over their profile and provided instructions on how to do this via the ORCID website. However, it was not particularly easy to explain the concept of how ORCID iDs were integrated with Europe PMC and authors expected to be able to do this from Europe PMC.

UK government and funder policies require researchers to publish their work Open Access. In 2014 a new requirement came into place which meant that all research articles which had been accepted for publication by a journal should be deposited in an institutional repository at the point of publication, and made available open access.

The brief was to help University of Cambridge researchers navigate the complex and rapidly changing publishing landscape and comply with policy requirements.

The challenge

Before I started on the project some user research had already been completed, to better understand the experience of academic publishing. A key findings was that academic researchers were not in touch with their institution at the point of publishing their work in a journal. And that the process of getting an article accepted for publication could be quite stressful. Therefore we knew it was going to be extremely challenging to persuade researchers to deposit a copy of what was known as their ‘author accepted manuscript’ in an institutional repository.

We wanted to better understand the diversity and pressure of academic roles, which could include running a research group, applying for grants, conducting research, writing papers, maintaining external collaborations, lecturing, supervision and pastoral care for students.

My role as user researcher

I led an ethnographic user research study, shadowing 10 academics for 2 days each. I planned and carried out the research collaboratively with two service designers. I created a template to capture our notes and we used small cameras to record images throughout the day.

I created ‘day in the life’ visuals and a research report to communicate insights to the team and stakeholders.

I created a service blueprint to help the team conceptualise the service.

I facilitated design studios and established ‘User Thursday’ to ensure regular testing of the designs. The design team used paper prototypes to rapidly iterate on the design.

I also created a grant funding application experience map to illustrate the experience of applying for grant funding.

My role as Product Manager

I led a team of 3 service designers and 3 developers to develop the service. I started by getting the team to define the problem to be solved using an adapted Lean UX canvas.

During development we continued testing designs iteratively with academic researchers.

I helped establish a new Open Access team to operationalise the service provided by the University Library team. This included configuring ZenDesk to manage support tickets from researchers.

I was responsible for outreach to University departments and presented to some senior academics.

Outcomes

In May 2014 we launched an Open Access pilot service which ensured the University complied with new policy requirements. Some academics in STEM disciplines like Physics, Maths and Astronomy were initially resistant to having to use the service to deposit their manuscripts. They were already posting their articles as preprints to a preprint server and didn’t like the duplication of effort. I took time to listen to their concerns and did some research into the preprint sever and found that the metadata attached to the articles would not meet the UK Government policy requirements. I liaised with the University in the US who managed the preprint server and explained the situation. As it was an issue only affecting UK researchers, it was not a high priority on their roadmap. By listening, responding, communicating clearly and adapting the service based on feedback from academic researchers I persuaded resistant stakeholders to get on board and use the service despite their initial concerns.