Adam Day

There’s a quote attributed to Ernest Rutherford: “That which is not measurable is not science. That which is not physics is stamp collecting”. I think his point was that a lot of scientific work is just

 documenting

things. In Rutherford’s day, there was a lot of exciting new

 creative

work happening in physics, so perhaps physics seemed special to him.

We need a chat about ChatGPT

TL;DR: Join me at ConTech Live to hear about a recent project with Open Credo to see if we could detect unusual co-authorships in a dataset created by Anna Abalkina. Sign up here! Papermilling has a few definitions which you see here and there.


(The?) 3 kinds of papermills

TL;DR: peer-review times appear to have been growing for a long time. The effects of COVID-lockdowns on peer-review are surprising.

A few months ago, I was invited to referee a research paper. So, I guess the editor thought that I was one of the 2 best people in the entire world to review this thing. That’s how it works, right? Flattered, I flagged the editor’s email to signal its importance and I got straight to work!

Is science getting slower?

The Papermill Alarm draws a line between papers which have the characteristics of papermill-products and those that don’t. Given data like that shown below, where red dots represent papermill-products and green dots represent normal papers, where should we draw that line?

Where to draw the line? Precision and recall

Recently, I gave a presentation on the APIs for Papermill Detection offered by Clear Skies Ltd. I also touch on a newer service called the Clear Skies Standard Report. More on that in future posts… :)  Here’s the video:  https://medium.com/media/ee5a04aaed9fc53d2748e64516178ebe/href Would you like to know more?

Video: APIs for Papermill detection (ConTech API Series)

Also… what is an API?

The Papermill Alarm API, is a service which you can send some article metadata to and which will return an alert telling you if the paper

 looks like

past papermill-products. Anyone can use it, but it definitely helps to have the support of an IT or data professional.

How to use the Papermill Alarm API

TL;DR: We can detect individuals with a high probability of being involved in milling papers. The question is: how should we respond?  A few years ago, my bike was stolen. It was an organised job. The thieves arrived after dark, cut a hefty lock clean off, and my bike disappeared silently forever. I imagined it dismantled and sold off into some vast black market network. Poor bike. The police admitted that they weren’t going to do anything.

Keystone: papermill detection with network analysis

TL;DR: The current version of the Papermill Alarm detects signals in 98.9% of the Hindawi retractions conducted over the last 12 months.** It’s never nice to see the harms caused by papermills, but it is good to see independent verification of the Papermill Alarm’s predictions.

Here’s a question: Why are there 2 multi-squillion-euro detectors at CERN’s Large Hadron Collider?

Predicting retractions with the Papermill Alarm

So, we’ve established that papermills like to use templates. We see templates in referee reports and in the text of cookie-cutter research papers. There’s an important insight here: Templates are used in legit academic behaviour as well as in industrial research fraud. So we need to be careful — we can’t assume that templates are necessarily a sign of research fraud.

Thank you for your warm work… earnestly

TL;DR: the rate of Papermill Alarm alerts is significantly higher in gold OA content than it is in subscription content. If you’re concerned about papermills, Clear Skies offers the Papermill Alarm: the world’s leading papermill detection tool. Get in touch for a demo. I’m a big believer in Open Science.

Papermills prefer Open Access

There’s this phrase used by some technologists: ‘epistemic security’. Epistemic security has to do with things like the spreading of misinformation on social media. I.e. if we spend enough time on social media reading unreliable information, how does that influence us? That’s a matter of epistemic security. Epistemic security is interesting because if you start a conversation about epistemic security at a party, they don’t invite you back.

The Papermill Alarm

TL;DR: There’s been a recent major upgrade to the private Papermill Alarm. It looks like we can predict the inclusion of journals on the Chinese Academy of Sciences (CAS) lists. You know how data scientists make predictions?

Cast a wide net

I once saw a brilliant presentation about how simple data analysis can detect credit card fraud**. The presentation showed a

 pattern

in how people use their credit cards. Given a large number of people who had been victims of credit card fraud, this pattern showed there was just 1 store in-particular where they had all used their cards. There was no

 observational evidence

of someone at that store stealing card details.

Patterns and Evidence (and how to find papermills before peer-review)

The first time I flew over the North Atlantic was quite an experience. Through the clouds, I could see some little white boats out sailing in the sea. It was puzzling: from 30,000 feet, those boats must have been huge for me to be able to see them at all.

How to find all the papermills (step 1)

This is Ralph. How tall is Ralph? It seems simple, you could just hold a ruler up to the screen. But when you look at the ruler and use it to measure Ralph, are you actually measuring

 Ralph

, or are you measuring the ruler and using that as a

 proxy

? How accurate is your measurement? Are you including fur in the measurement? What if Ralph were to stand on his hind legs, like a mighty bear — how tall would he be then?

The definition of plagiarism

Last week, a paper I wrote on the subject of peer-review fraud was published in the journal

 Scientometrics

(free link here, preprint here)

 .

It was an interesting project to work on. I found a lot of examples where one referee would write a report during peer-review and then another referee would write an identical report in some other peer-review of some other paper.

Papermills in peer-review

This post is about The Papermill Alarm: an API for detecting potential papermill-products. There’s a field of study called ‘stylometry’ where we look at the statistical properties of someone’s writing and use that to model their ‘style’. People write in idiosyncratic ways.

The Papermill Alarm: Patterns in PubMed

Stories by Adam Day on Medium

*TL;DR: The current version of the Papermill Alarm detects signals in
98.9% of the Hindawi retractions conducted over the last 12 months.\*\*
It's never nice to see the harms caused by papermills, but it is good to
see independent verification of the Papermill Alarm's predictions.*

Here's a question: Why are there 2 multi-squillion-euro detectors at
CERN's Large Hadron Collider? Why not save some money and just build 1?
[It's because you need one detector to independently verify what the
other is
detecting](https://home.cern/science/experiments#:~:text=Having%20two%20independently%20designed%20detectors,for%20focussing%20on%20specific%20phenomena).
Independent verification (or *reproducibility*) is a cornerstone
of science.

Peer-review is a relatively new invention. It's a system for checking
the scientific content of research. It is imperfect and it is
fundamentally based on trust.

Peer-review *isn't* a system for checking for scientific fraud. If
someone makes up their results, then peer-review isn't supposed to catch
that. *It might*, but it's not realistic to expect someone to reproduce
the results in peer-review because it might not be possible (e.g. in
cases where you would need a squillion euros to rebuild the experiment).
That means that, to some extent, reviewers have to take authors' claims
on faith.

It's because publishers run the peer-review process that they have
become the de facto gatekeepers of science. Now that we have clear
widespread scientific misconduct, publishers are thrust into the role of
research-cops tasked with policing, detective work, and even mopping up
the crime-scenes.

Publishers are part-responsible for research integrity, so that's fair
to an extent. It just means building capabilities outside of traditional
peer-review. But I think it's also fair to say that, compared with other
responsible parties: authors, institutions, funders, editors etc
publishers seem to cop a disproportionate share of the flak when things
go wrong. Again --- there's a big difference between peer-reviewing
science and detecting fraud.

I'm not the first to say it, but the solution to improving research
integrity is for all of these parties to work together and find ways to
adapt to the problem *together*. The problem isn't research integrity;
*there is no research in a fake paper.* On the other hand, the solution
*is* integrity. Honesty, accuracy and reproducibility could all go a
long way.

Hindawi's work to clean up the papermill problem in their journals is
laudable. They showed integrity in admitting mistakes and rectifying
them. They have made significant steps so far. It can't have been fun or
motivating work, but it was the responsible thing to do and that should
be recognised.

At Clear Skies, what we want is to catch as many cases as possible prior
to peer-review so that publishers can focus on quality service instead
of having to constantly hunt for fraud. Hindawi's retractions give us
some interesting insights there.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*LtEmJ4_ZLe_k6F7trN4LWQ.png" />
</figure>

The current version of the Papermill Alarm detects signals in **98.9%**
of the Hindawi retractions conducted over the last 12 months.\*\* It's
never nice to see the harms caused by papermills, but it is good to see
independent verification of the Papermill Alarm's predictions.

If you are interested in using the Papermill Alarm, either through your
peer-review platform, our API, or our webapp, [please get
in touch](https://clear-skies.co.uk/contact/).

#### Appendix: Data caveats

\*\* In the interest of responsible science, it's worth making a few
nerdy points about the data.

- The Papermill Alarm raises the alarm when the probability of a paper
 being connected to papermilling is significant. [There's a balancing
 act here which was described in a previous blog
 post](https://clearskiesadam.medium.com/where-to-draw-the-line-precision-and-recall-3c39a131c14b).
 So, while we find specific signals in 98.9% of Hindawi's retractions,
 we would trigger alerts in 90.2% of them. At that level, given that
 most of the retractions are concentrated in special issues containing
 multiple papers, I expect that we would catch all of those issues.
- The Papermill Alarm actually learns from retractions automatically. So
 in order to get the above figures, I had to remove Hindawi's
 retractions from the inputs to stop the Papermill Alarm from
 predicting something it had already seen.
- These figures come from a version of the tool that has not yet been
 released to the API, or webapp, so your mileage may vary if you are
 using those tools until the updates are released.
- When we talk about 'prediction' here, we are saying that the Papermill
 Alarm could find something that it hadn't seen before, but I should be
 clear that the retractions happened before the current version of the
 Papermill Alarm was created.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=08aadd1f3a98){width="1"
height="1"}

#### **Also... what is an API?**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*LJsC8JddbGR0xF8phQ_-yA.jpeg" />
<figcaption>API keys come in many forms. Source:
wiki commons</figcaption>
</figure>

The Papermill Alarm API, is a service which you can send some article
metadata to and which will return an alert telling you if the paper
*looks like* past papermill-products. Anyone can use it, but it
definitely helps to have the support of an IT or data professional.

(By the way, API stands for "application programming interface", but I
think my definition is clearer!)

APIs like this are used all over the web to transmit data between
systems. I'd like to tell you that using an API is as easy as using a
typical webform, and I'll describe a simple method for that in a moment.
But the truth is that it can be even easier: it can be completely
automated.

The **easiest** way to use any API is a 3 step process:

1. Walk into the IT department of the publishing company you work for
2. Shout loudly "HELLO! DOES ANYONE HERE KNOW HOW TO WORK AN API?"
3. Observe hands being raised and pick the one you like best.

*As a backup option, you might subtly rustle a pack of chocolate
digestives, or jammie dodgers. Your IT team will know what that means.*

Seriously though, using an API is second-nature to a lot of IT
professionals. Publishers use APIs routinely to interact with Crossref,
ORCID, ROR and others. So there should be someone in any organisation
who can help. The Papermill Alarm even comes with code examples in
multiple languages.

But we can actually use it without any code at all.

#### Before we start

Let's sign up for a subscription to the Papermill Alarm API. You'll
notice there are 2 options.

1. POST Batches
2. POST Single documents

If you just want to try the Papermill Alarm out, then let's go with
**single documents**. If you are experienced with APIs, then you will
want batch processing. Batches are faster and I'd recommend this option
for someone with experience of using APIs.

#### Checking an individual document (no coding required!)

Now simply

- navigate to [the RapidAPI page for the Papermill
 Alarm](https://rapidapi.com/clear-skies-clear-skies-default/api/papermill-alarm)
- click the 'single documents' option on the left side

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*bPj4kd8oVZRNW8o2tV3mtw.png" />
</figure>

- under 'Request Body', select 'Body' and edit the document in there so
 that it has 3 keys: "id", "title" & "abstract"

This might give us a document like this

 {
 “id”: “your_document_id”,
 “title”: “This is not a title of a paper”,
 “abstract”: “This is just an example piece of text. Not a real abstract.”
 }

> Note that the format here is 'JSON' and it is quite strict. We must
> use double quotes, **not** single quotes. There are colons between
> each 'key' and 'value' and then there's a comma at the end of each
> line EXCEPT the last one!

Click 'test endpoint'.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*5M3DQPbqN9WsWefU2qM8fA.png" />
</figure>

That was easy wasn't it?

<figure>
<img
src="https://cdn-images-1.medium.com/max/736/1*mZNbhde09kKiBN2ZazmdXQ.png" />
</figure>

#### Checking documents in a fast, automated way

But this could be a lot of work for a large publishing house with 100s
or 1000s of new submissions coming in each day. How should we use the
Papermill Alarm API if we have a lot of documents to check?

This is actually exactly what the API is designed for and, while there
wasn't much work in the above single-document check, it is quite
straightforward to check as many documents as we want in a completely
automated way.

Imagine eating breakfast every morning smugly aware that a computer has
already automatically flagged papermill-products coming into your
work queue.

I'm providing demo Python code for this in [a github
repository](https://github.com/ad48/papermill_alarm_get_started). This
example will show you how to check a large dataset. If you have a data
source for your new submissions it should be straightforward to set up a
scheduled task to check those and alert you when red alerts are
triggered.

My example is intended for someone with experience of Python. But note
that RapidAPI also provides code examples in a range of languages to
suit the skills of any particular IT professional.

If you'd like to hear more about scholarly APIs for papermill detection,
you should join me for [this ConTech Live API series
presentation](https://www.contech.live/api). See you there!

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=719b8b3b8253){width="1"
height="1"}

> Dissecting a joke is a bit like dissecting a frog. No one needs to
> know what's inside and the frog dies. --- Jimmy Carr

Last week, a paper I wrote on the subject of peer-review fraud was
published in the journal *Scientometrics* ([free link
here](https://rdcu.be/cVeAZ),
[preprint here](https://arxiv.org/abs/2202.03310))*.*

It was an interesting project to work on. I found a lot of examples
where one referee would write a report during peer-review and then
another referee would write an identical report in some other
peer-review of some other paper. (I wasn't the first person to observe
this behaviour, but I think this was the first method to detect it
automatically.)

If you look at all the data, you start to see networks of these
individuals copying peer-review comments. At least one of these clusters
is definitely the work of a paper mill (see if you can guess which one!)

<figure>
<img
src="https://cdn-images-1.medium.com/max/748/1*FYnTk1JQMAlYCBrV7a0hdQ.png" />
<figcaption>Each triangular node represents a referee account. Each red
link shows that 2 referees have, at some point, written identical
comments in peer-review.</figcaption>
</figure>

I think of this as 'tip-of-the-iceberg' analysis because only a subset
of paper-mill accounts and papers will ever show up when we use
this method.

Be that as it may, there is a huge bonus here. Say we find one referee
account duplicating a single comment from that big papermill cluster. In
that case, we can investigate that referee's publication history, the
other papers they've reviewed, the authors who have recommended them to
review, their co-authors and so on. So a single result can be the
first domino.

<figure>
<img
src="https://cdn-images-1.medium.com/max/726/1*JZ_4Lel5UL2fwuxcfPbRhA.png" />
<figcaption>With a bit of coaxing, we get a vast number of user accounts
to investigate. This is the same diagram as above once we add some more
search methods and connect author and co-author accounts. Authors are
green nodes, co-authors are grey. There’s some noise here, so a little
bit of simple graph analysis lets us prioritise the accounts with the
most connections. <a href="https://rdcu.be/cVeAZ">Details in
the paper</a>.</figcaption>
</figure>

### When to share

I may have laboured this point excessively in past blog posts, but one
thing that bothered me a lot in working on this paper was whether to
share the results of the work at all. Simply put: sharing *any*
misconduct investigation details publicly has the potential to educate
organised research fraudsters on how to circumvent detection. It's
therefore possible that sharing the method undermines it. Methods are
like frogs.

Publishers often advertise their plagiarism detection tools on their
submission pages. I'm sure that this is an effective deterrent some of
the time, but I wonder if it is also partly responsible for the rise of
paraphrasing in plagiarism (because plagiarists learned that plagiarism
detectors can't detect paraphrasing terribly well).

On the other hand, one might argue that bad actors will inevitably learn
about and find their way around any detection method eventually. It's
not clear what difference advertising it makes.

And there are the positive benefits to sharing:

- Even if fraudsters work around this detection method, doing so is
 *harder* than simply copy/pasting review comments as they were
 doing before.
- Sharing the method is one of the only ways to get it used broadly. It
 won't have much effect if only one publisher uses it.
- One thing that can't be worked around is the data already in
 publishers' systems. This analysis can be run retrospectively and
 suspect accounts can be flagged and investigated.
- Even if a method is well-known, there's always a future, naïve, new
 generation of fraudsters who don't know about it.

What's the 'right' thing to do? As with all ethical matters: it's
debatable. But our decision was that, considering the above, publishing
the method would move the needle in the right direction.

### This brings me to a somewhat bigger question. {#this-brings-me-to-a-somewhat-bigger-question.}

I spent many years working as a journal editor and, when I found a case
of suspected misconduct, I would follow a well-defined procedure to
investigate it.

- First, discuss the case carefully with various stakeholders.
- Then, as a rule, contact the authors and show them the evidence that
 they had committed that misconduct.

We would never *accuse* in that first message. We would simply show the
authors the evidence and ask for an explanation. I'll tell you now, that
if I ever presented an author with evidence like that, I was 100% sure
that they had done something wrong and that the evidence proved it
clearly. (Nevertheless, I would receive some wonderfully creative
excuses!)

But here's the problem: we can all agree that it's right and fair to
show someone the evidence if they are being accused of wrongdoing. But,
what if you are not dealing with an author? What if you are dealing with
an organisation that mills thousands of fake papers every year? In that
case, the evidence you send them is just teaching them how to improve
their business and you are not communicating with the 'author' at all.

So, there's the dilemma.

- We don't always know if we are dealing with a real author, or a fake
 account controlled by a papermill.
- So do we present the evidence and risk harming our chances of
 detecting fraud in the future?
- Or don't present the evidence and risk labelling someone as 'guilty'
 without giving them a fair opportunity to see or respond to the
 evidence?

Having thought about this a lot, I think the only practical option is to
aim to *never* share any information with suspected papermills. How
that's done (and particularly how to do it *ethically*) is, as I say:
debatable.

As for methods: it depends on how fragile the method is. If it's easy to
work around a detection method, I'm certainly not going to share it. If
it's hard to work around it, then we might *move the needle* by sharing
the method particularly if it's easy to use.

Right now, the methods exist to detect the overwhelming majority of
papermill products.

- Some of those methods should never see the light of day.
- Some of those methods might be published.
- Others might simply be shared privately.

#### Here are some examples of methods I think should be shared!

My recent paper (mentioned above):

- [Exploratory analysis of text duplication in peer‑review reveals
 peer‑review fraud and paper mills](https://rdcu.be/cVeAZ)

Some recent blog posts

- [How to find all the
 papermills](https://publisherad.medium.com/how-to-find-all-the-papermills-step-1-f00c624facb1)
- [The Papermill Alarm: patterns
 in Pubmed](https://publisherad.medium.com/the-papermill-alarm-patterns-in-pubmed-b89db7feba9d)

Any questions, get in touch!

- [Twitter](https://twitter.com/AdamSci12)
- [LinkedIn](https://www.linkedin.com/in/%F0%9F%98%B7-adam-day-45529a17/)
- [Clear Skies](https://clear-skies.co.uk/) (my side-business --- not
 related to my work at SAGE)

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c507b12aa4ac){width="1"
height="1"}

The first time I flew over the North Atlantic was quite an experience.
Through the clouds, I could see some little white boats out sailing in
the sea. It was puzzling: from 30,000 feet, those boats must have been
huge for me to be able to see them at all. Suddenly, the clouds were
gone and, through clear skies, I could see a really big 'little white
boat' and I understood at once what I had been looking at.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*DePhN8tzJjWHH38gUksYLA.jpeg" />
<figcaption>Image CC-BY: wiki commons <a
href="https://commons.wikimedia.org/wiki/File:Sarqarput-strait-iceberg-aerial.jpg">https://commons.wikimedia.org/wiki/File:Sarqarput-strait-iceberg-aerial.jpg</a></figcaption>
</figure>

With papermills, people often say that they only see 'the tip of the
iceberg'. (It's almost a cliché!) If fake papers were easy to detect,
then forgers would just learn to get better at writing them. I guess
that's what actually happened over time and it's why we need to be
super-careful about sharing detection methods.

This is why I've thought very carefully about making this post (and why
I've no intention of publishing steps 2, 3 etc).

#### Here's what we know:

- [Papermills use duplicate submission
 routinely](https://medium.com/p/21ebd625ad40)
- [Detecting duplicate submission
 is easy](https://youtu.be/6yA3phww3m8?t=1579)
- [Retrospective analysis is a
 necessity](https://medium.com/p/9ab21e84d0c9)

With this in mind, it's actually very simple for publishers to see
the iceberg.

1. Make your submission dates visible in Crossref.

It might not be trivial to do, but it shouldn't be hard or
controversial. Most journals make their submission dates visible in
article XML, so this wouldn't make any data visible that isn't already,
it just makes it easier to work with. In fact, several publishers are
doing this already.

2\. Search for your submissions in Crossref.

This is also not hard to do. The results will show you when your
submissions were published by others and (with the above step having
been done) when they were submitted to those other publishers.
Conveniently, there are some pre-built options that will make doing this
trivial (see 'Article Tracking Options' below!).

3\. Finally, if you find a duplicate submission, contact the publisher
of that duplicate submission and show them: the submission and decision
dates as well as the full-text. For them to confirm duplicate submission
requires only a quick manual comparison of the manuscripts. It's much,
**much** quicker than a full misconduct investigation.

### What does this give us?

Well, there are 2 things:

- All of the papermill submissions (at least all of those that have been
 using duplicate submission --- there are exceptions).
- Any other authors using duplicate submission as a hedge.

Isn't this great? No fancy AI, no need to actually detect and prove the
fakery. Here is all the evidence you need to prove a paper has breached
your terms of service. Furthermore, if we take this to its logical
conclusion(which might be an up-front cross-publisher full-text check
made at the submission of any article), you can see that there really
isn't any way around it. The paper-mills will have to stop sending the
same paper to different publishers concurrently. That will slow them
down considerably. It's also why I'm comfortable sharing this publicly.

But how to tell the two apart --- papermills and 'other' duplicates?
Well, I'm not going to share that here, but what I will say is that it
isn't hard. The iceberg is obvious once you see it through clear
skies ;)

At this point, it's up to you how you proceed. You might want to analyse
the data to learn more about papermills. You might want to conduct
misconduct investigations. You might see opportunities to collaborate
with other publishers. After all, this method does require that we all
work together.

### Article tracking options

1. The SAGE Rejected article tracker.

This was a project that started out as [a blog post that I wrote some
years ago in my spare
time](https://publisherad.medium.com/rejected-article-tracking-with-the-crossref-api-22d75a874904).
It later became [an open source project from SAGE Publishing with its
own peer-reviewed
paper](https://joss.theoj.org/papers/10.21105/joss.03348). It's free and
it will get the job done (if a little slowly). Note that, in this case,
it's important to track all articles, this includes accepted articles,
new submissions and (importantly!) abandoned submissions as well as
rejected articles. Remember how [I said that we sometimes see the same
paper published in 2 places](https://medium.com/p/9ab21e84d0c9)? This is
how to find those cases.

2\. [My own "Article
Tracker" API](https://rapidapi.com/clear-skies-clear-skies-default/api/article-tracker/details).

This is part of a side-project (not related to my work at SAGE) and I
hadn't intended to make it publicly available, but after seeing the
value of this tool in detecting papermills, here it is. It's the best
tool of its kind as far as I know. I hope that it helps with
this problem.

The Article Tracker has the following features:

- Fast: in testing, I've been able to track around 500 articles
 per minute.
- Accurate: the current accuracy is around 96--98% on test data. There
 is a planned upgrade which will push accuracy even higher.
- Does duplicate submission checking (as described above) out of
 the box.
- I've set pricing at a low level where I think any publishing house
 should be able to easily afford to search for all of their submissions
 as well as historic submissions without running into limits.

This blog post is actually the first time I've advertised the Article
Tracker's existence, so you might consider this to be a soft-launch of
the service. Any questions, comments or feedback would be very welcome.

- [You can contact me through the Article Tracker's parent
 company](https://clear-skies.co.uk/contact/) or
- [find me on Twitter](https://twitter.com/AdamSci12).

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f00c624facb1){width="1"
height="1"}