Blog

Peer Review Week is dedicated to discussing all things peer review, and this year’s theme focuses on peer review and the future of publishing.

Ensuring that peer review keeps pace with, and in some cases drives, ongoing developments in the wider academic and scholarly publishing community is key to enabling all parts of the process to work together efficiently.

In honor of Peer Review Week, we spoke to Ashley Farley, Program Officer of Knowledge and Research Services and Open Access Lead at the Bill and Melinda Gates Foundation, to get her thoughts on this topic.

Before we get into some of the challenges we see in traditional peer review, what do you think is the importance of peer review for researchers and wider stakeholders?

For researchers, I think peer review supports natural conversation, iteration, and the evolution of research. It’s a mechanism for feedback and critique of the work and is often done in a way that strives to overcome potential bias, which is an important endeavor in disseminating knowledge.

For other stakeholders, whether it’s policymakers or the general public, the concept (I’m careful not to say execution here) of peer review acts as a “trust signal” – that others have read, evaluated, and/or tested the conclusions and that any potential issues or additional experiments have been completed to produce a final article that will (hopefully) stand the test of time.

On the whole, peer review is often seen as a safeguarding mechanism against poorly executed research or outright fraudulent claims.

You were careful to make the distinction between the concept of peer review and the execution of it. What would you say were the challenges in current peer review models for research, and what are the impacts of this?

I think that the biggest challenge in peer review is finding reviewers who will commit to the full process of peer review and put in the volunteer time as an expert in their field.

As more research is published, as more research is funded, as the “publish or perish” monster needs to be fed, it’s becoming hard for journals to have manuscripts quickly reviewed.

It’s near impossible to scroll through social media and not see a thread from a researcher or editor lamenting over the struggle to have a paper move through the system in a timely manner or to find reviewers who will get the task done.

This is not sustainable, yet little has been done to alleviate issues or reimagine peer review to accommodate current, and anticipate future, needs.

I try to participate in peer review as much as possible – when invited or through the ASAPbio Preprint Peer Review work – and I am often surprised at how inefficient and performative it can feel.

Most articles on the topics I work in (librarianship, open access, open science, open data, funder policies) could be published without peer review, and most comments are forward-looking in future developments. Yet, the articles can get stuck in a circle of review that drags on for months.

Additionally, as more research is conducted on research and scholarly publishing itself, we are learning and assessing potential risks. Some examples include:

• Peer review often operates as a gatekeeping exercise that assesses the novelty of a paper for a specific journal, rather than evaluating the scientific soundness of the work (high impact journals that reject over 90% of submissions – I’m looking at you). This serves little purpose to the research community other than protecting the journal brand, and most of these papers are not reproducible.
• Many types of outputs that funders and their grantees want widely available are not considered appropriate for journal publication and, thus, peer review. Examples include case studies, how-to guides, commentaries, perspectives, policy documents, data notes, method articles etc. These outputs often can be more valuable to communities that are not steeped in traditional academic practices.
• Research comparing a preprint and the subsequent journal version of record has found that while the article may be improved to some degree, there is not a significant difference between versions.
• While preprints may not be peer reviewed, they are cost-effective to post and share while being journal agnostic. The community can then broadly and quickly assess the work.
• Have you seen Retraction Watch? What is the baseline for successful peer review that doesn’t lead to a retraction?

We have to ask the question, then, what purpose is current peer review really serving?

And, as there is such a strong potential for bias behind close reviews, is the system actually effective at being a safeguarding measure against poor research?

What would you say were the impacts of these challenges for research funders?

I feel funders are at a crossroads with peer review.

Peer review is considered critical for most research and, as a result, many funder open access (OA) policies so far have focused on the journal’s Version of Record (VoR), as these have been peer reviewed in the traditional sense of inviting peer reviewers from a pool of individuals.

Over time, Plan S and the Rights Retention Strategy have shifted this from the VoR to the Author Accepted Manuscript (AAM) to give authors more autonomy and ownership over their work. This means that if VoRs can’t be published open access, then at least we can aim for open access for AAMs, as these have also been peer reviewed in the same way.

However, as rising costs of OA publication are making this harder to achieve, funders are beginning to look at preprints.

But if funders have a preprint-only policy, with pre-prints not subjected to the same peer review processes as articles published in journals, does that mean that peer review is less important for funders?

It’s clear that there are many issues within the current peer review processes. What do you see as the future of peer review?

I think that as OA policies continue to develop and publishing innovations at a large scale stagnate, we need to consider innovative ways forward for peer review.

I’ve been thinking a lot about what I’ve been calling “organic peer review”, which is a peer review that is provided without invitation, often post publication. These are the reviews you will find on PubPeer, preprint servers, social media, or other platforms.

Further facilitating this type of peer review, I believe, will be extremely beneficial and sustainable. The papers needing to be peer reviewed will be, and the system will be more equitable if moving away from invitation only.

Additionally, the scope of who can participate in peer review must expand from using a database to select several considered ‘experts’ in a given discipline. There are many more groups that can provide useful feedback and insight.

As research becomes more complex, peer review must reflect this and may need multiple people to assess different components of an article – for example, a statistician to review the analysis, a librarian to review the literature/systematic review, a developer to review/run the code, etc.

Early career researchers have a lot of insight to provide in these areas and are often overlooked in peer review invites.

These all sound like positive steps forward in peer review. What do you think needs to change to get there?

I think the main thing that needs to happen is that stakeholders across the academic publishing sector, including publishers and editors, need to become comfortable with testing, iteration, and a bit of risk.

Most would agree that the status quo isn’t sustainable, yet there is little evidence of testing potential solutions and thinking outside of the box.

Similarly, publishers and funders could better recognize and support (mainly financially) platforms that provide post-publication peer review of articles.

Gates Open Research is already moving towards some elements of this future peer review approach you’ve identified. How can the peer review model at Gates Open Research contribute to overcoming these challenges and help meet the needs of future publishing?

I truly believe that the peer review model at Gates Open Research should be the standard for several reasons.

Post-publication review

The article is published first, with formal, invited peer review taking place after publication.

This operates in a similar way to a preprint and means that the article is not held up from being widely shared – an author doesn’t have to wait months or years to be able to share their work.

I also believe that organic post-publication peer review will, over time, prove to be more effective in identifying issues within articles than the current process, by including the views of many different specialists.

Open peer review

Open peer review provides the reader with the conversation that typically happens behind the scenes. This adds value and understanding to the various article versions, and I cannot believe that this important context isn’t shared more frequently.

Similarly, when reviews are opened and reviewers named, it is harder for reviewers to produce unduly negative or biased feedback than it is behind closed doors.

While there can be power imbalances causing concerns for early career researchers who worry that critiquing senior researchers can lead to retaliation, I feel the only way to fix this is to enforce open peer review and call out bad behavior.

Open peer review at Gates Open Research also includes open user commenting, alongside open reports and open reviewer identities, which means people across the community can be involved in the discussion, not just those who have been invited as reviewers.

Incentivizing peer review and providing opportunities for co-reviewing

Having open peer review reports means that each peer report can be cited independently, and reviewers can use these reports to demonstrate involvement in this important activity. This can help incentivize the practice of peer review.

Additionally, there is the opportunity to co-review reports and name multiple authors in a single review, which helps provide equal opportunities for early career researchers who may otherwise be overlooked as reviewers, while adding more transparency and trust to the review.

 Thank you for your insights Ashley!

Peer reviewing with Gates Open Research

If you’d like to be part of the change, you can find out more about peer review at Gates Open Research and express your interest in becoming a reviewer yourself.

If you’d like to benefit from this peer review approach for your own work, find out more about publishing your Gates-funded work with the platform.

• Sign up to our newsletter
• Follow us on Twitter