Björn Brembs

A few [years
ago](https://bjoern.brembs.net/2021/03/scholarly-publishing-in-three-cartoons/),
I came across a cartoon that seemed to capture a particular aspect of
scholarly journal publishing quite well:

<figure class="wp-block-image">
<img src="https://i.imgur.com/POng9wL.jpeg" decoding="async"
alt="https://i.imgur.com/POng9wL.jpeg" />
</figure>

The academic journal publishing system sure feels all too often a bit
like a sinking boat. There are many leaks, e.g.:

-- a reproducibility leak  
-- an affordability leak  
-- a functionality leak  
-- a data leak  
-- a code leak  
-- an interoperability leak  
-- a discoverability leak  
-- a peer-review leak  
-- a long-term preservation leak  
-- a link rot leak  
-- an evaluation/assessment leak  
-- a data visualization leak  
etc.

A more recent leak that has sprung up is a papermill leak. What is a
'papermill'? Papermils are organizations that churn out journal articles
that are made to look superficially like research articles but basically
only contain words without content. How big of a problem are papermills
for science?

In a recent
[article](https://www.theguardian.com/science/2024/feb/03/the-situation-has-become-appalling-fake-scientific-papers-push-research-credibility-to-crisis-point),
the Guardian cites [Dorothy
Bishop](https://en.wikipedia.org/wiki/Dorothy_V._M._Bishop) with regards
to papermills:

> In many fields it is becoming difficult to build up a cumulative
> approach to a subject, because we lack a solid foundation of
> trustworthy findings. And it's getting worse and worse.

The article states that something on the order of 10,000 articles a year
being produced by papermills poses a serious problem to science. These
numbers most certainly are alarming! The article also cites [Malcolm
Macleod](https://en.wikipedia.org/wiki/Malcolm_Macleod):

> If, as a scientist, I want to check all the papers about a particular
> drug that might target cancers \[...\], it is very hard for me to
> avoid those that are fabricated. \[...\] We are facing a crisis.

OK, challenge accepted, let's have a look at cancer research, where the
[reproducibility rate](https://elifesciences.org/articles/67995) of
non-papermill publications is just under 12%, so we'll round it to that
figure. PubMed lists about one million papers (excluding reviews) on
cancer in the last 5 years:

<figure class="wp-block-image size-large">
<a
href="https://pubmed.ncbi.nlm.nih.gov/?term=%28cancer%29+NOT+%28Review%5BPublication+Type%5D%29&amp;filter=years.2019-2024&amp;timeline=expanded&amp;sort=date"><img
src="https://bjoern.brembs.net/wp/wp-content/uploads/2024/02/image-1024x519.png"
class="wp-image-2806" data-fetchpriority="high" decoding="async"
srcset="https://bjoern.brembs.net/wp/wp-content/uploads/2024/02/image-1024x519.png 1024w, https://bjoern.brembs.net/wp/wp-content/uploads/2024/02/image-300x152.png 300w, https://bjoern.brembs.net/wp/wp-content/uploads/2024/02/image-768x389.png 768w, https://bjoern.brembs.net/wp/wp-content/uploads/2024/02/image-50x25.png 50w, https://bjoern.brembs.net/wp/wp-content/uploads/2024/02/image.png 1267w"
sizes="(max-width: 1024px) 100vw, 1024px" width="1024"
height="519" /></a>
</figure>

If the sample result of 12% were representative, this would mean that
the last 5 years in cancer research produced about 880,000 unreliable
publications, or about 176,000 per year. And that's just cancer. Let's
also pick psychology, where [replication
rates](https://www.science.org/doi/10.1126/science.aac4716) were
published as 39% in 2015. 2015 is a long time ago and psychology as a
field really went to great lengths to address the practices giving rise
to these low rates. Therefore, let's assume things got better in the
last decade in psychology, so after 4-5 years, maybe 50% replication was
achievable. Searching for psychology articles yields about 650,000
non-review articles in the last 5 years:

<figure class="wp-block-image size-large">
<a
href="https://psycnet.apa.org/search/results?id=a806f48d-d0ea-3dc9-8efe-c874fda82743"><img
src="https://bjoern.brembs.net/wp/wp-content/uploads/2024/02/image-1-1024x184.png"
class="wp-image-2807" decoding="async"
srcset="https://bjoern.brembs.net/wp/wp-content/uploads/2024/02/image-1-1024x184.png 1024w, https://bjoern.brembs.net/wp/wp-content/uploads/2024/02/image-1-300x54.png 300w, https://bjoern.brembs.net/wp/wp-content/uploads/2024/02/image-1-768x138.png 768w, https://bjoern.brembs.net/wp/wp-content/uploads/2024/02/image-1-50x9.png 50w, https://bjoern.brembs.net/wp/wp-content/uploads/2024/02/image-1.png 1369w"
sizes="(max-width: 1024px) 100vw, 1024px" width="1024"
height="184" /></a>
</figure>

This amounts to about 65,000 unreliable psychology articles per year.

So according to these very (very!) rough estimates, just the two fields
of cancer research and psychology together add more than a million
unreliable articles to the literature every five years or so. Clearly,
those are crude back-of-the-envelope estimates, but they should be
sufficient to just get an idea about the orders of magnitude we are
talking about.

If the numbers hold that about 2 million articles get published every
year, just these two fields would together amount to a whopping 10% of
unreliable articles. Other major reproducibility projects in the social
sciences and economics yield reproducibility rates in these fields of
about 60%. Compare these numbers to a worst case scenario of all
papermills together producing some 10k unreliable articles a year. If
the scholarly literature really were a sinking boat, fighting papermills
would be like trying to empty the boat with a thimble, or plug the
smallest hole with a cork.

When will we finally get ourselves a [new
boat](https://royalsocietypublishing.org/doi/10.1098/rsos.230206)?


A few years ago, I came across a cartoon that seemed to capture a particular aspect of scholarly journal publishing quite well:  The academic journal publishing system sure feels all too often a bit like a sinking boat.

Messaggi di Rogue Scholar

How reliable is the scholarly literature?