Adam Day

There’s a quote attributed to Ernest Rutherford: “That which is not measurable is not science. That which is not physics is stamp collecting”. I think his point was that a lot of scientific work is just

 documenting

things. In Rutherford’s day, there was a lot of exciting new

 creative

work happening in physics, so perhaps physics seemed special to him.

We need a chat about ChatGPT

TL;DR: Join me at ConTech Live to hear about a recent project with Open Credo to see if we could detect unusual co-authorships in a dataset created by Anna Abalkina. Sign up here! Papermilling has a few definitions which you see here and there.


(The?) 3 kinds of papermills

TL;DR: peer-review times appear to have been growing for a long time. The effects of COVID-lockdowns on peer-review are surprising.

A few months ago, I was invited to referee a research paper. So, I guess the editor thought that I was one of the 2 best people in the entire world to review this thing. That’s how it works, right? Flattered, I flagged the editor’s email to signal its importance and I got straight to work!

Is science getting slower?

The Papermill Alarm draws a line between papers which have the characteristics of papermill-products and those that don’t. Given data like that shown below, where red dots represent papermill-products and green dots represent normal papers, where should we draw that line?

Where to draw the line? Precision and recall

Recently, I gave a presentation on the APIs for Papermill Detection offered by Clear Skies Ltd. I also touch on a newer service called the Clear Skies Standard Report. More on that in future posts… :)  Here’s the video:  https://medium.com/media/ee5a04aaed9fc53d2748e64516178ebe/href Would you like to know more?

Video: APIs for Papermill detection (ConTech API Series)

Also… what is an API?

The Papermill Alarm API, is a service which you can send some article metadata to and which will return an alert telling you if the paper

 looks like

past papermill-products. Anyone can use it, but it definitely helps to have the support of an IT or data professional.

How to use the Papermill Alarm API

TL;DR: We can detect individuals with a high probability of being involved in milling papers. The question is: how should we respond?  A few years ago, my bike was stolen. It was an organised job. The thieves arrived after dark, cut a hefty lock clean off, and my bike disappeared silently forever. I imagined it dismantled and sold off into some vast black market network. Poor bike. The police admitted that they weren’t going to do anything.

Keystone: papermill detection with network analysis

So, we’ve established that papermills like to use templates. We see templates in referee reports and in the text of cookie-cutter research papers. There’s an important insight here: Templates are used in legit academic behaviour as well as in industrial research fraud. So we need to be careful — we can’t assume that templates are necessarily a sign of research fraud.

Thank you for your warm work… earnestly

TL;DR: the rate of Papermill Alarm alerts is significantly higher in gold OA content than it is in subscription content. If you’re concerned about papermills, Clear Skies offers the Papermill Alarm: the world’s leading papermill detection tool. Get in touch for a demo. I’m a big believer in Open Science.

Papermills prefer Open Access

There’s this phrase used by some technologists: ‘epistemic security’. Epistemic security has to do with things like the spreading of misinformation on social media. I.e. if we spend enough time on social media reading unreliable information, how does that influence us? That’s a matter of epistemic security. Epistemic security is interesting because if you start a conversation about epistemic security at a party, they don’t invite you back.

The Papermill Alarm

TL;DR: There’s been a recent major upgrade to the private Papermill Alarm. It looks like we can predict the inclusion of journals on the Chinese Academy of Sciences (CAS) lists. You know how data scientists make predictions?

Cast a wide net

I once saw a brilliant presentation about how simple data analysis can detect credit card fraud**. The presentation showed a

 pattern

in how people use their credit cards. Given a large number of people who had been victims of credit card fraud, this pattern showed there was just 1 store in-particular where they had all used their cards. There was no

 observational evidence

of someone at that store stealing card details.

Patterns and Evidence (and how to find papermills before peer-review)

The first time I flew over the North Atlantic was quite an experience. Through the clouds, I could see some little white boats out sailing in the sea. It was puzzling: from 30,000 feet, those boats must have been huge for me to be able to see them at all.

How to find all the papermills (step 1)

This is Ralph. How tall is Ralph? It seems simple, you could just hold a ruler up to the screen. But when you look at the ruler and use it to measure Ralph, are you actually measuring

 Ralph

, or are you measuring the ruler and using that as a

 proxy

? How accurate is your measurement? Are you including fur in the measurement? What if Ralph were to stand on his hind legs, like a mighty bear — how tall would he be then?

The definition of plagiarism

Last week, a paper I wrote on the subject of peer-review fraud was published in the journal

 Scientometrics

(free link here, preprint here)

 .

It was an interesting project to work on. I found a lot of examples where one referee would write a report during peer-review and then another referee would write an identical report in some other peer-review of some other paper.

Papermills in peer-review

This post is about The Papermill Alarm: an API for detecting potential papermill-products. There’s a field of study called ‘stylometry’ where we look at the statistical properties of someone’s writing and use that to model their ‘style’. People write in idiosyncratic ways.

The Papermill Alarm: Patterns in PubMed

Stories by Adam Day on Medium

There's a quote attributed to Ernest Rutherford: "That which is not
measurable is not science. That which is not physics is stamp
collecting". I think his point was that a lot of scientific work is just
*documenting* things. In Rutherford's day, there was a lot of exciting
new *creative* work happening in physics, so perhaps physics seemed
special to him. On the other hand, it was a mean-spirited thing to say
about other fields of science and I'm sure he felt very silly when he
was handed the Nobel prize in chemistry.

Generative AI is all the rage right now isn't it? Essentially, we can
now use AI models to do *creative* tasks, like painting a picture.

Do you remember that famous series of paintings that Claud Monet did
called "computer nerds hard at work"?

<figure>
<img
src="https://cdn-images-1.medium.com/max/512/1*ag7wUHny722vK6beW0UdfQ.png" />
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/512/1*GVoYZVm-u9uQuxJvjhWKIg.png" />
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/512/1*U9V_-9NXaDl0O3CbDU0dwQ.png" />
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/512/1*T5AMW56GTO2GSe6eW5I1VA.png" />
</figure>

No? Wikipedia says that Monet passed away in 1926. That means he was
dead at the time these computers were invented and can't possibly have
painted them, but they certainly look like Monet's work, don't they? If
I had wanted to fake images like this 12 months ago, I would have needed
canvas, oils and a lot of talent with a brush. Today, I just prompted
Stable Diffusion to do it for me. No coding skills required. It
took seconds.

Speaking of coding, I was looking for a very specific piece of code
recently. After half an hour searching on Google and StackOverflow I was
ready to give up searching and write it myself, but then I asked ChatGPT
to do it for me. ChatGPT wrote exactly what I needed. Again, it
took seconds.

ChatGPT is a chatbot based on a *Large Language Model* called GPT-3. You
can ask ChatGPT to write just about anything. Poems, stories, homework
solutions, research papers... anything!

The value of new AI models for **creative** tasks like this is immense.
Images and text and even video can be generated rapidly with
increasingly authentic quality standards.

I put it to you that when you write a research paper, that *isn't*
creative work. I mean, it obviously is to an extent, but the purpose of
a research paper is to document something. It's the stamp collecting
part of science and that's an important distinction.

If I ask ChatGPT for text for an original research paper, it will give
it to me. But the text won't be based on any real novel piece of
research, it will be made up and it will be either derivative, false, or
both. It will look real and [a recent study by Catherine Gao at
Northwestern University in Chicago,
Illinois](https://www.nature.com/articles/d41586-023-00056-7) showed
that scientists were often unable to distinguish real scientific
abstracts from those written by ChatGPT.

By the way, if I prompt Stable Diffusion for fake scientific images,
it'll cough those up too.

So, AI can already produce believable scientific fakes that are hard to
detect. Worse, the people building these AI applications might not even
be aware of the extent and dangers of [organised
research fraud](https://medium.com/@publisherad/how-papermills-work-generally-c080bb9fff43).

Through my company, Clear Skies, I've already been working on a couple
of methods that should be able to detect fraudulent ChatGPT-generated
content. But I'm in 2 minds as to whether it's a good idea to develop
this capability because there's no good reason that AI should be
producing this kind of content in the first place.

- AI shouldn't be trained to generate documentary material at all.
- APIs for these tools should also classify prompts and output to
 recognise when they are being abused for fraudulent or harmful
 purposes (and then decline to give output to a user).

I hope that doesn't sound like wishful thinking. Controls like these
actually do exist on both ChatGPT and Stable Diffusion. So I am
optimistic that, as the people building these applications receive
feedback, they make adaptations like those above to avoid creating
harmful outputs.

In the meantime, we are already seeing policies on artificially
generated content. StackOverflow [has temporarily banned AI-generated
content until such time as it can determine suitable guidelines
for use](https://meta.stackoverflow.com/questions/421831/temporary-policy-chatgpt-is-banned).

The future of this problem for Schol-Comms is complex. There is a lot to
consider:

- Should publishers ban AI-generated content like StackOverflow
 has done?
- Or should guidelines be more nuanced --- like should AI-generated
 content be treated like quotations where the non-original content in a
 paper is clearly set apart from the rest of the text? What about AI
 autocomplete? Should that be banned or quoted, or is that different?
- Do the copyright licenses on scientific papers allow them to be used
 for training generative AI? (And what should those licenses allow?
 It's not like they were written with this use case in mind.)

It's a tough one and I'm sure it'll take time to figure out the best way
to handle this. What is clear, though, is that there is plenty
to discuss!

There are a couple of addenda worth making here. There are other AI
use-cases that can be trained on research papers. E.g.
question-answering. That's where we can give an open-ended question like
"what's the best treatment for COVID-19" and then an AI model can find
all the best papers on that topic and generate a response to the
question based on those. That's a brilliant thing to do and so there's
definitely a good case for training models for some tasks.

It's also worth mentioning Meta's recently released Galactica model.
Galactica was supposed to enable tasks like those described above.
However: there was clear potential to use the model for research fraud
and it had a few other issues (e.g. it produced nonsense from time to
time --- if we want scientific answers, that's not very helpful). The
dangers were clear and [I was relieved to see Meta kill Galactica just
days after its
release](https://www.technologyreview.com/2022/11/18/1063487/meta-large-language-model-ai-only-survived-three-days-gpt-3-science/).
There's still potential for a project like that, but it's great to see
decisive action from Meta to address the risk.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72ab43bd217){width="1"
height="1"}