Adam Day

There’s a quote attributed to Ernest Rutherford: “That which is not measurable is not science. That which is not physics is stamp collecting”. I think his point was that a lot of scientific work is just

 documenting

things. In Rutherford’s day, there was a lot of exciting new

 creative

work happening in physics, so perhaps physics seemed special to him.

We need a chat about ChatGPT

TL;DR: Join me at ConTech Live to hear about a recent project with Open Credo to see if we could detect unusual co-authorships in a dataset created by Anna Abalkina. Sign up here! Papermilling has a few definitions which you see here and there.


(The?) 3 kinds of papermills

TL;DR: peer-review times appear to have been growing for a long time. The effects of COVID-lockdowns on peer-review are surprising.

A few months ago, I was invited to referee a research paper. So, I guess the editor thought that I was one of the 2 best people in the entire world to review this thing. That’s how it works, right? Flattered, I flagged the editor’s email to signal its importance and I got straight to work!

Is science getting slower?

The Papermill Alarm draws a line between papers which have the characteristics of papermill-products and those that don’t. Given data like that shown below, where red dots represent papermill-products and green dots represent normal papers, where should we draw that line?

Where to draw the line? Precision and recall

Recently, I gave a presentation on the APIs for Papermill Detection offered by Clear Skies Ltd. I also touch on a newer service called the Clear Skies Standard Report. More on that in future posts… :)  Here’s the video:  https://medium.com/media/ee5a04aaed9fc53d2748e64516178ebe/href Would you like to know more?

Video: APIs for Papermill detection (ConTech API Series)

Also… what is an API?

The Papermill Alarm API, is a service which you can send some article metadata to and which will return an alert telling you if the paper

 looks like

past papermill-products. Anyone can use it, but it definitely helps to have the support of an IT or data professional.

How to use the Papermill Alarm API

TL;DR: We can detect individuals with a high probability of being involved in milling papers. The question is: how should we respond?  A few years ago, my bike was stolen. It was an organised job. The thieves arrived after dark, cut a hefty lock clean off, and my bike disappeared silently forever. I imagined it dismantled and sold off into some vast black market network. Poor bike. The police admitted that they weren’t going to do anything.

Keystone: papermill detection with network analysis

TL;DR: The current version of the Papermill Alarm detects signals in 98.9% of the Hindawi retractions conducted over the last 12 months.** It’s never nice to see the harms caused by papermills, but it is good to see independent verification of the Papermill Alarm’s predictions.

Here’s a question: Why are there 2 multi-squillion-euro detectors at CERN’s Large Hadron Collider?

Predicting retractions with the Papermill Alarm

So, we’ve established that papermills like to use templates. We see templates in referee reports and in the text of cookie-cutter research papers. There’s an important insight here: Templates are used in legit academic behaviour as well as in industrial research fraud. So we need to be careful — we can’t assume that templates are necessarily a sign of research fraud.

Thank you for your warm work… earnestly

TL;DR: the rate of Papermill Alarm alerts is significantly higher in gold OA content than it is in subscription content. If you’re concerned about papermills, Clear Skies offers the Papermill Alarm: the world’s leading papermill detection tool. Get in touch for a demo. I’m a big believer in Open Science.

Papermills prefer Open Access

There’s this phrase used by some technologists: ‘epistemic security’. Epistemic security has to do with things like the spreading of misinformation on social media. I.e. if we spend enough time on social media reading unreliable information, how does that influence us? That’s a matter of epistemic security. Epistemic security is interesting because if you start a conversation about epistemic security at a party, they don’t invite you back.

The Papermill Alarm

TL;DR: There’s been a recent major upgrade to the private Papermill Alarm. It looks like we can predict the inclusion of journals on the Chinese Academy of Sciences (CAS) lists. You know how data scientists make predictions?

Cast a wide net

I once saw a brilliant presentation about how simple data analysis can detect credit card fraud**. The presentation showed a

 pattern

in how people use their credit cards. Given a large number of people who had been victims of credit card fraud, this pattern showed there was just 1 store in-particular where they had all used their cards. There was no

 observational evidence

of someone at that store stealing card details.

Patterns and Evidence (and how to find papermills before peer-review)

The first time I flew over the North Atlantic was quite an experience. Through the clouds, I could see some little white boats out sailing in the sea. It was puzzling: from 30,000 feet, those boats must have been huge for me to be able to see them at all.

How to find all the papermills (step 1)

This is Ralph. How tall is Ralph? It seems simple, you could just hold a ruler up to the screen. But when you look at the ruler and use it to measure Ralph, are you actually measuring

 Ralph

, or are you measuring the ruler and using that as a

 proxy

? How accurate is your measurement? Are you including fur in the measurement? What if Ralph were to stand on his hind legs, like a mighty bear — how tall would he be then?

The definition of plagiarism

Last week, a paper I wrote on the subject of peer-review fraud was published in the journal

 Scientometrics

(free link here, preprint here)

 .

It was an interesting project to work on. I found a lot of examples where one referee would write a report during peer-review and then another referee would write an identical report in some other peer-review of some other paper.

Papermills in peer-review

This post is about The Papermill Alarm: an API for detecting potential papermill-products. There’s a field of study called ‘stylometry’ where we look at the statistical properties of someone’s writing and use that to model their ‘style’. People write in idiosyncratic ways.

The Papermill Alarm: Patterns in PubMed

Stories by Adam Day on Medium

This post is about [The Papermill
Alarm](https://rapidapi.com/clear-skies-clear-skies-default/api/papermill-alarm):
an API for detecting potential papermill-products.

There's a field of study called 'stylometry' where we look at the
statistical properties of someone's writing and use that to model their
'style'. People write in idiosyncratic ways. So, for example, I often
start sentences with 'so, for example' when I could more succinctly say
'for example'. The stylistic properties of written work can therefore,
potentially, give away the identity of the author. [Famously, it's been
shown that J K Rowling shares stylistic traits with Robert
Galbraith](https://www.scientificamerican.com/article/how-a-computer-program-helped-show-jk-rowling-write-a-cuckoos-calling/),
something which is confirmed by the fact that both of them, without
exception, are the same person.

Stylometry methods might seem dated these days, but don't discount them.
They have their uses!

I recently made a blog post about 2 of the main roles I see in
papermilling --- Agents and Forgers. Forging fake research papers is a
skill. You have to know enough of the science to be able to write a
believable fake. (It's depressing. Think about it: the people forging
those fakes must have had real scientific ambitions at some point.)

The thing is, though, that if you can write a believable fake paper
about cancer genetics, you probably can't write a believable fake about
geology, or about engineering. So, what do you do? (Keep writing fakes
about cancer genetics, of course!) The result is that we find unusual
clusters of samey papers.

### Let's take a look at PubMed

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*Jr9E1K-nAJzk_LH_JIIFAw.png" />
</figure>

What you're seeing here is a map of PubMed. More specifically, this is
the titles of PubMed papers put through 2 processes:

- The Univeral Sentence Encoder is used to turn each title into a 1024
 dimensional vector. That's too many dimensions for me. If I try to
 visualise 1024 dimensions in my head, it makes my eyes water.
- A process called 'UMAP' is used to compress those vectors into simple
 2-dimensional x and y coordinates. Much better!

Maps like this are easy to make and a lot of fun to explore. They do
lose a lot of data in that compression stage and so, while the map can
be used for this kind of rough visual exploration, it shouldn't be
thought of as a tool for precision science. Also, keep in mind that we
are just looking at *titles* here. The Papermill Alarm models abstracts
as well.

You might recognise some of the features of PubMed on this map.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*Y4YZ49eiSDrgdtjroprxMA.png" />
</figure>

The bright green data-points are the titles of [papers that have been
identified as suspicious in various lists of papers by Elisabeth Bik,
Smut Clyde and
others](https://docs.google.com/spreadsheets/d/1zKxfaqug4ZhwHyGzslF38pFyC8xtU8lzmmOFMGYITDI/edit#gid=0).
Note that these cases are concentrated in specific regions of PubMed.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*B4gNCQZyHA1uMUjykCZACw.png" />
<figcaption>There are 2 new clusters here at the foot of the map. One
concerns polymers and the other one mathematics. These 2 don’t appear to
overlap PubMed.</figcaption>
</figure>

*I really can't overstate the importance of the kind of work that people
like Elisabeth Bik, Smut Clyde, several anonymous PubPeer commenters
(and countless referees and journal editors behind the scenes) do in
carefully reviewing these papers for flaws!*

Wouldn't it be nice to flag papers which share the stylistic features of
those groups? So that, if a paper fitting one of those specific regions
comes into your peer-review system, you can be alerted that it looks a
bit like a papermill? That would be good wouldn't it? That's what [The
Papermill
Alarm](https://rapidapi.com/clear-skies-clear-skies-default/api/papermill-alarm) does!

Here is a map showing a random sample of what The Papermill Alarm thinks
look like papermills (in red). Note the clear correlation with those
green dots.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*oHngLlm7tW3_wbocQNjbiA.png" />
</figure>

### The key takeaways

You can see now that there is clear agreement between the red alerts and
the papers that have already been picked out by hand.

There's also definite scope to expand the Papermill Alarm to fields
outside of PubMed. E.g. those green clusters at the bottom.

Finally, the Papermill Alarm is also detecting *growth* in papers that
trigger red alerts. That's how I can be confident that you will see
alerts for more papermill papers by the Papermill Alarm in the future.
(I should be clear that I don't intend this as a quantitative estimate
of the scale of papermilling.)

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*OTSy5DD_PGMIKcyZ" />
</figure>

Any questions? Get in touch!

- [Twitter](https://twitter.com/AdamSci12)
- [LinkedIn](https://www.linkedin.com/in/%F0%9F%98%B7-adam-day-45529a17/)

### Appendix: Some caveats about the visualisations

Reminder: this is just for illustrative purposes.

- The 50,000 blue dots, which represent all of PubMed are also only a
 small proportion of the total. There are millions of papers in PubMed.
- Red alerts make up only around 1% of the total PubMed output. This
 might not be obvious from looking at the above because everything is
 subsampled. Red dots are also deliberately brighter than blues.
- The 2,000 red alerts (red dots) you're seeing are only a small
 (randomly sampled) percentage of all the papers picked up by the
 Papermill Alarm.
- There are around 3,000 green dots.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b89db7feba9d){width="1"
height="1"}