#PeerReviewWeek: Why we need a FAIR Principles for Peer Review?

As we enter Peer Review Week 2025, we would like to propose the use of FAIR (Findable, Accessible, Interoperable and Reusable) principles for scientific peer review, using these to make the process more scalable, efficient and also better equipped for the increasing use of AI in publishing.

Today marks the start of Peer Review Week 2025 (PRW 2025), an annual event celebrating the vital role peer review plays in ensuring the quality and integrity of research. Regular readers and our authors will know of our "Open Science Peer Review" efforts in making peer review more transparent and accountable, with mandated Open Peer Review (see our Editorial) and increasing integration with preprints and many third party review sharing platforms. We have acknowledged and participated in these PRW celebrations over the years, using it as an occasion to announce milestones in our Open Science peer review process such as DOIs for peer reviews in 2015, experiments and webinars with AcademicKarma (a pioneering preprint peer review platform) in 2017, integration with the bioRxiv/medRxiv TRiP (Transparent Review in Preprints) platform in 2021, and our integration with the Sciety platform in 2023 that aggregates preprint peer reviews and helps move our GigaByte journal towards the 'Publish, Review, Curate' (PRC) peer-review model developed by eLife.

This Open Science peer review workflow system we have built at GigaByte combines transparent open peer review, mandated use of preprints, and integrated sharing of the reviews back to the linked preprints using the TRiP and Sciety platforms. With our open peer reviews giving credit to the hard work our reviewers do, and the peer reviews and an additional "Editorial endorsement" statement on the preprint providing additional insight and provenance to the preprints as well. We have recently created an explainer video to better explain how this works and you can see it here.

The theme of Peer Review Week this year is "Rethinking Peer Review in the AI Era", and making our peer reviews more discoverable and machine readable in this manner has helped train Sciety's associated Artificial Intelligence (AI) tools to more accurately find related evaluated preprints. This demonstrated to us some of the benefits of making peer review content AI-ready, and seeing this started to make us think about the FAIR Guiding Principles for scientific data, which were developed very much with AI-readiness in mind.

What are the FAIR Guiding Principles for scientific data?

The FAIR principles for scientific data management and stewardship are guidelines for research data management, FAIR standing for Findable, Accessible, Interoperable, and Reusable. Going beyond just Open Data, to giving some broad guidance on how enhance the reusability of data, and particularly thinking about how best to help machines to automatically find and use the data. Since these were published in 2016 (with the GigaScience Editor in Chief being one of the authors), these principles have been particularly influential, the focus on the AI-readiness of data being especially topical and prescient. In the decade since it's formulation (see the blog on the 10th anniversary last year), FAIR has taken a life of it's own, and this FAIR approach has been applied to many other areas such as research software (FAIR4 Research Software) and computational workflows (FAIR workflows and FAIR Digital Objects).

Why do we think Peer Reviews also need to be FAIR?

Growing public mistrust in science, coupled with the enormous growth in the number of journals and papers has put increasing strain and scrutiny on peer review. What many have dubbed as a Peer Reviewer Crisis, being able to find the number of peer reviewers to keep up with the volume of new papers is becoming increasingly untenable and some feel has already reached breaking point. This isn't helped by the way we traditionally carry out peer review being very inefficient. Anonymous, closed peer review, coupled with papers often getting rejected and resubmitted to many journals before publication means that most of this peer review effort is wasted and never used. Portable peer review, where peer reviews can be moved between journals reduces this duplication and wastage, and much of the rationale for the eLife 'Publish, Review, Curate' model of peer review is to enhance efficiency in this manner. Framing these efforts through the FAIR principles can help to increase this efficiency and scalability of peer review even further, and also helps address the trust issue through increased transparency and scrutiny of the peer review system.

How thinking about peer review in this manner could fit with FAIR could work as follows:

F – Findable – Making peer reviews findable increases transparency and trust, and increases efficiency by reducing wastage of peer reviews. Opening up peer review is key to this, and Publons (now part of Clarivate) was a pioneering example of a platform that allowed researchers to track and showcase their peer review contributions (and we were one of the first journals to work with them when they launched). Giving these open peer reviews DOIs, providing rich metadata, and indexing them in relevant scholarly databases allows them to be discovered beyond the many silos they currently sit inside.

A- Accessible – Like F, making peer reviews accessible increases transparency and trust, and increases efficiency by reducing wastage of peer reviews. Open licensing (all our reviews are CC-BY) reduces additional barriers for reuse, as does linking them to author names/ORCID IDs and associated preprints.

I – Interoperable – There have been many attempts to make peer reviews portable between journals, which has been relatively easy between journals inside the same publishing house, slightly harder but still possible within particularly collaborative fields (see the Neuroscience Peer Review Consortium, of which we are a member) or during emergencies such as COVID (see the C19RapidReview Initiative), but the many third party (mostly commercial) services attempting to do this across publishers have mostly fallen by the wayside. The rise of preprint peer review and overlay journals have been encouraging step towards this interoperability, but this process needs to get more automated and seamless between preprints, journals and indexes. Efforts such as the JATS4R "Peer review materials" best practice guidelines published in 2021 assist publishers in structuring and tagging their peer reviews to work towards this interoperability, and the DocMaps framework under development takes this a step further through defining a machine-readable protocol for sharing and discovery.

R – Reusable – Once peer reviews have got more Findable, Accessible and Interoperable; the final barrier is convincing journal Editors to trust what they say and actually reuse them. Different journals have different Editorial thresholds and standards, and they may ask different questions and have different structures of peer reviewer forms, but it shouldn't be too hard to standardise the format of peer reviews across journals (tackling the Interoperability issue). Cultural challenges are more of an issue than technical challenges for Reusability, as getting Editors to give up control on reviewer selection and trust external peer-reviews has been very tricky. Open Peer Review reduces barriers here as Editors will always want to scruitinise the background and experience of the reviewers used, but in a world of identity theft and paper mills there is also paranoia that the reviews are real and were written by the people who say they are from. Verification by ORCID helps, and there are new digital identity and personhood verification tools such as VeriMe that will help here through even more stringent levels of identity-verification and trust.

At GigaScience Press we've been very happy to consider and reuse externally derived peer reviews from third-party preprint recommendation and review services like Peer Community In (we've published papers coming through PCI Ecology and PCI Genomics) ResearchHub, and our early experiments with Preprint.Space that included using it for real-time transparent conference paper review. But in doing this we've always emailed the reviewers to verify their identity and check the email addresses given are real, and to double check they are happy with our strict open peer review policies. If the provenance of the reviews can be guaranteed, COIs are declared and checked; the handling Editor can then decide whether to use the reports or not, and invite additional reviewers if necessary. This is the approach GigaScience Press has taken, but until other publishers are happy to do the same, the lack of will to reuse peer reviews will continue to be a major bottleneck. Review Commons has managed to get 28 affiliate journals across a number of publishers to link up with preprint peer review, but participation is currently very field specific (slowly growing out of developmental and cell biology) and been quite exclusive to a small number of non-profit publishers, and (at least from our experience when we have tried to join) have been slow to welcome and allow other journals to participate. This experiment has at least has shown peer review of preprints can work across publishers, but this process needs to be more streamlined, and field and publisher agnostic to be able to be scalable enough to make a dent in tackling the peer review crisis.

Rethinking Peer Review in the AI (and FAIR) Era

Going back to the topic of Peer Review Week 2025, AI is already having massive effects on scientific publishing, and provides both challenges and opportunities in peer review. Where people have debated whether AI is the answer to the peer review problem, or may be the problem itself. Keeping on top of peer review when we have surpassed the five million annual published papers mark, and AI assistance in research (carrying out the research, coding research software, and also writing it up as papers) is potentially what will make the peer review crisis reach breaking point (Mark Hahnel proposing a model to calculate when this may be). Improving AI-readiness was a key driver in the development and uptake of the original FAIR principles, and as we explain here this goal can also be very relevant to peer review.

Where publication volume is increasingly outpacing reviewer capacity, using AI to improve the tools assisting Editors find peer reviewers is an obvious (and non-controversial) use of the technology. Being one of the very first journals to work with Publons we helped them test very early versions of these types of tools, but they really didn't work when there was so little training data. Fast forward a decade, with much bigger pools of peer reviews and papers to train on, alongside huge improvements in the algorithms (particularly LLMs) these tools now seem to be working much effectively. Our Editors have used the Clarivate Web of Science Reviewer Locator (which evolved from the Publons tools) and the Prophy.ai semantic search reviewer tools in helping find peer reviewers and have found the results of both useful. There is a growing range of these tools available, and the more targeted and accurate they are will save much wasteful spamming of unsuitable referees. Although a big challenge is still finding Early Career Researchers (ECRs, e.g. Postdocs and PhD students) who are fantastic reviewers (having the most up-to-date experience and wider diversity of perspectives), but may not have the previous track record in publications and reviews that these tools find them from. Schemes like the ASAPbio Preprint Reviewer Recruitment Network (of which we participated in) have tried to address this gap by proactively getting ECRs to share and post preprint peer reviews. Moving towards FAIR Peer Review will assist all of these efforts, increasing the visibility and discovery of ECR peer review outputs, increasing and widening the pool of training data, and improving it's structuring and metadata to allow these AI tools to work much better and accurately.

Moving all this information into the commons will also also promote the development of open source approaches, rather than have multiple publisher controlled tools built upon on many walled gardens of less accurate proprietary data. From including all our preprint peer-reviews into the Sciety platform we've already found their "Related articles" and Looker Studio journal dashboard useful for commissioning submissions from preprints that are very on-topic for our journals. Seeing first hand that the more data we've trained the algorithms with, the better targeted the results that come back are.

A More FAIR Rise of the Machines

The most controversial (but potentially most promising) use of AI in peer review is using it in the review process itself. If major challenge for peer review problem is scalability, and automated validation tools are already commonly used for assessing many types of data, then AI should an obvious way to augment, improve and scale up peer review. But as peer review in most cases is anonymous, confidential, and deals with information not yet in the public domain, the Research Integrity community (such as COPE) has been particularly concerned about leaking of confidential information and the risks of "idea theft". Particularly as the business models of the AI companies are unclear, input data is taken by them to be re-used for model training, and this process is opaque as the algorithms are proprietary black boxes. There have also been reports of attempted hacking of AI-conference paper submissions using "prompt injection", sneaking secret messages into papers in an effort to trick AI-review tools into giving them positive peer-reviews.

Because of this 59% of the 78 top medical journals with guidance on the matter have banned AI use in peer review. While these companies offer opt-outs for sharing data back with them, there needs to be ironclad security (e.g. if researchers host offline LLMs on their own computers) and much more transparency (see our work on AI standards for example) before these tools can be truly trusted. As with the increasing adoption and usage of AI in other parts of the research cycle (see our polices on use of LLMs in writing for example), these AI-generated assessments also need to have humans check and take responsibility for what is shared and used. There has been a huge rush to develop these types of reivew tools (see Chris Leonard's Scalene blog to keep on top of progress here), and our initial thoughts from testing some of these concurs with others that the outputs currently seem very uniform and generic, and works for some article types better than others. But as we've seen with AI-tools in other areas, with improvements to the algorithms and training data quality and quantity these tools will very rapidly improve.

FAIR Peer Review integrating Open Peer Review and preprints (enabling authors to have already claimed priority for new findings) greatly reduces the data security challenges as everything is already in the public domain and there is no confidential information to leak. FAIR Peer Review should also improve the accessibility, breadth and quality of training data, in turn improving the quality and speeding the development of AI peer review tools. Making the development of these tools and algorithms more transparent and open should help tackle the trust issue, and also aid the development of open source and community driven tools and approaches which again should be more trustworthy than AI companies and the big publishers controlling everything from many untransparent silos and black box proprietary algorithms.

Moving to FAIR Peer Review in this manner should greatly improve scalability of peer review. As publication outputs continue to skyrocket in an unsustainable manner in the long term it may not completely solve the Peer Review Crisis, but it will certainly buy us more time. It also doesn't address the fundamental question of the value and cost of journals mediated peer review and if it is truly worth the expense and effort we put into it, but being more systematic and transparent in this manner should have the side effect of improving the quality of what is currently a rushed and ad hoc process.

This is just a very roughly sketched out proposal trying to make people think about why a FAIR principles for peer review would be useful rather than a properly developed guideline. To actually implement these ideas, as with the original FAIR guiding principles for scientific data management and stewardship, these principles would need to be mapped and laid out in much greater detail. Different stakeholder perspectives are needed to provide input and insight, and working groups of these would need to elaborate and define what exactly is required and fits into the four core desiderata (F-A-I-R). And then throw it out to the wider community to respond and give feedback. While we haven't done any of those things here, as the stated aim of Peer Review week is to "Explore, Engage and Elevate" and we hope this discussion piece at least addresses that.

See more on Peer Review Week 2025 here: https://peerreviewweek.net/

References

Edmunds SC. Peering into peer-review at GigaScience. Gigascience. 2013 Jan 24;2(1):1. https://doi.org/10.1186/2047-217x-2-1

Wilkinson M et al., (2016). The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 https://doi.org/10.1038/sdata.2016.18

Chue Hong NP et al., (2022). FAIR Principles for Research Software (FAIR4RS Principles) (1.0). Zenodo. https://doi.org/10.15497/RDA00068

Tropini C et al., Time to rethink academic publishing: the peer reviewer crisis. mBio. 2023 Dec 19;14(6):e0109123. https://doi.org/10.1128/mbio.01091-23

How to cite this post: Edmunds S. #PeerReviewWeek: Why we need a FAIR Principles for Peer Review?GigaBlog. 2025. https://doi.org/10.59350/dprsg-bzm06