Critical perspectives in metascience and the philosophy of science relating to open science, science reform, and the replication crisis.

Mark Rubin

During multiple testing, researchers often adjust their alpha level to control the familywise error rate for a statistical inference about a joint union alternative hypothesis (e.g., “

 H1

or

 H2

”). However, in some cases, they do not make this inference.

Redundant Multiple Testing Corrections

In an article published last week in Synthese, philosopher of science Pekka Syrjänen asked “does a theory become better confirmed if it fits data that was not used in its construction versus if it was specifically designed to fit the data?” The first approach is called

 prediction,

and the second approach is called

 accommodation

. The debate over the epistemic advantages of prediction and accommodation has been bubbling away for

Prediction vs Accommodation: Which is Better and When?

Putting Popper to work

Replication, falsification, and the crisis of confidence in social psychology

Why Replication Is Overrated

Imre Lakatos

The role of replication in psychological science

Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis

Inference to the best explanation: A neglected approach to theory appraisal in psychology

Why Are Snowy Owls White and Why Have They Evolved Distinct Sexual Color Dimorphism? A Review of Questions and Hypotheses

Ceteris paribus clauses, closure clauses and falsifiability

The swan genome and transcriptome, it is not all black and white

Appraising and Amending Theories: The Strategy of Lakatosian Defense and Two Principles that Warrant It

Three Fragments From a Sociologist's Notebooks: Establishing the Phenomenon, Specified Ignorance, and Strategic Research Materials

Are psychologists appraising research properly? Some Popperian notes regarding replication failures in psychology.

Ceteris Paribus Laws

“Repeated sampling from the same population?” A critique of Neyman and Pearson’s responses to Fisher

The Preregistration Prescriptiveness Trade-Off and Unknown Unknowns in Science: Comments on Van Drimmelen (2023)

Making replication mainstream

Abstract   Popper’s (1983, 2002) philosophy of science has enjoyed something of a renaissance in the wake of the replication crisis, offering a philosophical basis for the ensuing science reform movement. However, adherence to Popper’s approach may also be at least partly responsible for the sense of “crisis” that has developed following multiple unexpected replication failures.

The Replication Crisis is Less of a “Crisis” in the Lakatosian Approach than it is in the Popperian and Naïve Methodological Falsificationism Approaches

Felipe Romero presented his work - “The conceptual origins of metascience: Fashion or revolution?” - in The Popper Seminar at the London School of Economics on 30th May 2023. Here, I highlight a few points that he raised and provide some comments along the way. ::: {#youtube2-AKWCCdK-jgc .youtube-wrap attrs=“{"videoId":"AKWCCdK-jgc","startTime":null,"endTime":null}” component-name=“Youtube2ToDOM”} Ein Fehler ist aufgetreten.

What's Special About Metascience?

Tom Hostler has just published an article titled “The invisible workload of open research” in the

 Journal of Trial and Error’s

special issue on the “Consequences of the scientific reform movement: Is the scientific reform movement headed in the right direction?” It’s an insightful and thought-provoking piece that uncovers the potential workload costs of open research, considers why these costs may be ignored by university management,

Open Science and Overwork

Research Registries: Facts, Myths, and Possible Improvements

Ensuring the quality and specificity of preregistrations

Comparing dream to reality: an assessment of adherence of the first generation of preregistered studies

Challenges for assessing replicability in preclinical cancer biology

Moving Beyond the "Replication Crisis": Understanding the Exploratory/Confirmatory Data Analysis Continuum

Aspiring to greater intellectual humility in science

Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling

Paths in strange spaces: A comment on preregistration

Preregistration Is Hard, And Worthwhile

A Call for Greater Modesty in Psychology and Cognitive Neuroscience

The effectiveness of preregistration in psychology: Assessing preregistration strictness and preregistration-study consistency

Researchers’ decision making: Navigating ambiguity in research practice

Researchers’ decision making: exercising discretion when navigating ambiguity in research practice 

Abstract   I discuss Van Drimmelen’s (2023) Metascience2023 presentation on researchers’ decision making during the research process.

The Preregistration Prescriptiveness Trade-Off and Unknown Unknowns in Science

The Centre for Open Science’s symposium on “Critical Perspectives on the Metascience Reform Movement” took place on 7

 th

March 2024. It was organised by Sven Ulpts, and it includes presentations by Bart Penders, Tom Hostler, Stephan Guttinger, Sarahanne Field, Nicole Nelson, and Berna Devezer.

Critical Perspectives on the Metascience Reform Movement

Sabina Leonelli’s new book – “Philosophy of Open Science” – will be published later this year. However, there is an open access preprint available on the PhilSci Archive here. In the book, Leonelli provides “a constructively critical reading” of the standard approach to open science which, she argues, is focused on sharing “objects” such as data and materials.

Sabina Leonelli’s “Philosophy of Open Science”

I really enjoyed Mel Andrews’ recent essay: "Philosophy in the Trenches and Laboratory Benches of Science" and its main point that “every laboratory needs a philosopher.” This point is made in the context of a concern about “the industrialisation of science”:  This concern about the industrialisation of science reminded me of something that statistician Ronald Fisher wrote many years ago when he argued that the Neyman-Pearson approach to

The Industrialisation of Science

An Intellectual Vocation
 

In his book “The Soul of a University” (2018), Chris Brink describes the story of G.H. Hardy, a Cambridge Mathematician whose principled stance on his academic research was that it had no practical use whatsoever: “No discovery of mine” Hardy proudly wrote, “has made, or is likely to make, directly or indirectly, for good or ill, the least difference to the amenity of the world”

Tom Hostler

A long-read guest essay by Tom Hostler on how the scientific reform movement aligns with John Ziman's concept of "post academic research".

Scientific reform, post-academic research, and academic identity

The December 2022 issue of the journal

 Industrial and Organizational Psychology

includes a very interesting discussion of the pros and cons of open science. There’s a target article titled “Open science, closed doors: The perils and potential of open science for research in practice” (Guzzo et al., 2022) and nine commentaries on this article. You can find all 10 papers here.

Opening Up Open Science to Epistemic Pluralism

In this new article, I consider questionable research practices in the field of metascience. A questionable metascience practice (QMP) is a research practice, assumption, or perspective that’s been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. I discuss 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science.

Questionable Metascience Practices

The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as

 p

-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view.

Type I Error Rates are Not Usually Inflated

An often overlooked source of the “replication crisis” is the tendency to treat the replication study as a definitive verdict while ignoring the statistical uncertainty inherent in both the original and replication studies. This simplistic view fosters misleading dichotomies and erodes public trust in science.

Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication

The Rules of the Game Called Psychological Science

Power failure: why small sample size undermines the reliability of neuroscience

Beyond Power Calculations

Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?

The sciences are different and the differences matter

Addressing Exaggeration of Effects from Single RCTs

Forcing a Deterministic View on Probabilistic Phenomena: Implications for the Replication Crisis

Preregistration Distinguishes Between Exploratory and Confirmatory Research?

Previous justifications for preregistration have focused on the distinction between “exploratory” and “confirmatory” research. However, as I discuss in this recent presentation, this distinction faces unresolved questions.

Acceptibility, Evidence, and Severity

The Statistical Crisis in Science

The case for formal methodology in scientific reform

Bayes and Beyond

Prediction Versus Accommodation and the Risk of Overfitting

Changes in the Problem of Inductive Logic

When and How to Deviate From a Preregistration

The benefits of preregistration and Registered Reports

How About Bust? Factoring Explanatory Power Back into Theory Evaluation

Severe Testing as a Basic Concept in a Neyman–Pearson Philosophy of Induction

Does preregistration improve the credibility of research findings?

The Costs of HARKing

Inconsistent multiple testing corrections: The fallacy of using family-based error rates to make inferences about individual hypotheses

Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests

A Bayesian perspective on severity: risky predictions and specific hypotheses

On the Use and Misuses of Preregistration: A Reply to Klonsky (2024)

Preregistration does not improve the transparent evaluation of severity in Popper’s philosophy of science or when deviations are allowed

Preregistration, Severity, and Deviations

A conference called “The promises and pitfalls of preregistration” was hosted by the Royal Society in London from 4th-5th March 2024. Here, I discuss the presentations by Chris Donkin and Stephan Lewandowsky, both of which consider some of the potential “pitfalls” of preregistration.

Critical Metascience