Mark Rubin

The Centre for Open Science’s symposium on “Critical Perspectives on the Metascience Reform Movement” took place on 7
<sup>
 th
</sup>
March 2024. It was organised by Sven Ulpts, and it includes presentations by Bart Penders, Tom Hostler, Stephan Guttinger, Sarahanne Field, Nicole Nelson, and Berna Devezer.

Critical Perspectives on the Metascience Reform Movement

Ross Mounce

This is just a quick post of appreciation for [PCI Registered
Reports](https://rr.peercommunityin.org/). I've recently joined the PCI
RR community as a
'[recommender](https://rr.peercommunityin.org/about/recommenders)'.

<figure class="wp-block-image size-full">
<a
href="https://rossmounce.co.uk/wp-content/uploads/2023/11/image.png"><img
src="https://rossmounce.co.uk/wp-content/uploads/2023/11/image.png"
class="wp-image-2279" loading="lazy" decoding="async" width="259"
height="194" /></a>
</figure>

  
One thing that spurred me to join is a rather unsatisfactory experience
I had as a peer-reviewer, reviewing a manuscript where the experimental
design was deeply insufficient. I won't reveal for which open access
journal I reviewed, or the full content of my peer review report but the
gist of it was (quoting from my report, my words):

> *"...this \[name of OA journal redacted\] manuscript has an inadequate
> sample size to draw conclusions from. Sampling only 152 journals out
> of a total population of \~4594 SCIE-indexed biology journals, equates
> to just a 3.3% sample size (152/4594). I see no reason why the authors
> couldn't have taken at least a 10% sample (over 459 journals). Prior
> work on open access citation advantage such as Ghane et al (2020) did
> their work on a sample of 1881 journals indexed in DOAJ (DOAJ at the
> time indexed a total of \~11,000 journals; Ghane et al (2020) achieved
> a greater than 10% sample size).*
>
> *Furthermore, the choice of how which journals were sampled was by
> their own account in a non-random and arbitrary manner. Students
> merely \_chose\_ a journal to download data on. This is highly
> subjective and prone to bias. As an example of one possible bias
> introduced: if I were a student, I might be tempted to choose a
> journal with a low article volume so that it would be less work/time
> to complete the assignment..."*
>
> Ross Mounce (2023)

TL;DR the experimental design was deeply flawed. Although I appreciate
the authors put a lot of work into it, the conclusions of their work
were just not valid, given the problems of their experimental design.

The experience as a reviewer was like arriving at the scene of a car
crash six months too late to stop the accident. To change this piece of
work into one with an acceptable experimental design at this point would
be a LOT of new work and might require throwing away much of the
existing data.

It's clear to me that a lot of researchers are now trying their hand at
metaresearch, research on research, journalology, and open access
studies --- whatever you call it. That's great. But I implore folks to
seriously consider getting their experimental design peer-reviewed
BEFORE they start their study, and *PCI Registered Reports* is one of
the communities that will provide that service for you, for free!

I would love to see more stage 1 registered reports in metaresearch, to
catch improperly designed studies before they get carried out,
written-up, and submitted. Reducing research waste AND increasing the
quality of research that ends-up getting published --- what's not to
like? It might save you a lot of anguish.


This is just a quick post of appreciation for PCI Registered Reports. I’ve recently joined the PCI RR community as a ‘recommender’.  One thing that spurred me to join is a rather unsatisfactory experience I had as a peer-reviewer, reviewing a manuscript where the experimental design was deeply insufficient.

Kudos to PCI Registered Reports

Putting Popper to work

Replication, falsification, and the crisis of confidence in social psychology

Why Replication Is Overrated

Imre Lakatos

The role of replication in psychological science

Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis

Inference to the best explanation: A neglected approach to theory appraisal in psychology

Why Are Snowy Owls White and Why Have They Evolved Distinct Sexual Color Dimorphism? A Review of Questions and Hypotheses

Ceteris paribus clauses, closure clauses and falsifiability

The swan genome and transcriptome, it is not all black and white

Appraising and Amending Theories: The Strategy of Lakatosian Defense and Two Principles that Warrant It

Three Fragments From a Sociologist's Notebooks: Establishing the Phenomenon, Specified Ignorance, and Strategic Research Materials

Are psychologists appraising research properly? Some Popperian notes regarding replication failures in psychology.

Ceteris Paribus Laws

“Repeated sampling from the same population?” A critique of Neyman and Pearson’s responses to Fisher

The Preregistration Prescriptiveness Trade-Off and Unknown Unknowns in Science: Comments on Van Drimmelen (2023)

Making replication mainstream

Abstract   Popper’s (1983, 2002) philosophy of science has enjoyed something of a renaissance in the wake of the replication crisis, offering a philosophical basis for the ensuing science reform movement. However, adherence to Popper’s approach may also be at least partly responsible for the sense of “crisis” that has developed following multiple unexpected replication failures.

The Replication Crisis is Less of a “Crisis” in the Lakatosian Approach than it is in the Popperian and Naïve Methodological Falsificationism Approaches

Messages de Rogue Scholar

Critical Perspectives on the Metascience Reform Movement

Kudos to PCI Registered Reports

The Replication Crisis is Less of a “Crisis” in the Lakatosian Approach than it is in the Popperian and Naïve Methodological Falsificationism Approaches