Mark Rubin

In this [new article](https://doi.org/10.36850/mr4), I consider
questionable research practices in the field of metascience. A
*questionable metascience practice* (QMP) is a research practice,
assumption, or perspective that's been questioned by several
commentators as being potentially problematic for metascience and/or the
science reform movement. I discuss 10 QMPs that relate to criticism,
replication, bias, generalization, and the characterization of science.
My aim is not to cast aspersions on the field of metascience but to
encourage a deeper consideration of its more questionable research
practices, assumptions, and perspectives.

<figure>
<img
src="https://cdn-images-1.medium.com/max/666/1*gUc7USykxjwSsshmLL51hQ.jpeg" />
</figure>

### **10 QMPs** {#qmps}

(1)️ **Rejecting or ignoring self-criticism**: Rejecting or ignoring
criticisms of metascience and/or science reform

(2)️ **Fast 'n' bropen scientific criticism**: A quick, superficial,
dismissive, and/or mocking style of scientific criticism

(3)️ **Overplaying the role of replication**: Assuming that replication
is essential to science, and that it indexes "the truth"

(4)️ **Unspecified replication rate targets**: Assuming a replication
rate is "too low" without specifying an "acceptable" rate

(5)️ **Metabias**: An unacknowledged bias towards explaining the
replication crisis in terms of researcher bias

(6)️ **A bias reduction assumption**: Focusing on selective reporting as
the primary form of researcher bias and assuming that it can be reduced
without increasing other forms of bias

(7)️ **Devaluing exploratory results**: Devaluing an exploratory result
as being more "tentative" than a confirmatory result without considering
other relevant issues (e.g., quality of associated theory, methods,
analyses, transparency)

(8)️ **Presuming QRPs are problematic**: Presuming that questionable
research practices (e.g., HARKing) are always problematic research
practices

(9)️ **A focus on knowledge accumulation**: Conceiving knowledge
accumulation as the primary objective of science without considering (a)
the role of specified ignorance or (b) different objectives in other
philosophies of science

\(10\) **Homogenizing science**: Focusing on specific approaches (e.g.,
quantitative methods; replicable effects) as "the scientific method"

### **A Caveat**

In my article, I stress that only *some* metascientists engage in *some*
QMPs *some* of the time, and that these QMPs may not *always* be
problematic. Research is required to estimate the prevalence and impact
of QMPs. In the meantime, I think that QMPs should be viewed as
invitations to ask questions about how we go about doing better
metascience.

<figure>
<img
src="https://cdn-images-1.medium.com/max/500/1*wEXB4y5pQWq9xASq52Jrew.jpeg" />
</figure>

### **The Article**

Rubin, M. (2023). Questionable metascience practices. *Journal of Trial
and Error.* <https://doi.org/10.36850/mr4>

### My Other Work in This Area

For my other work in the area of metascience, please see:
[https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis](https://www.blogger.com/blog/post/edit/7326905493335343666/6062434439681064559#)

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cceefa31ce46){width="1"
height="1"}


In this new article, I consider questionable research practices in the field of metascience. A
<em>
 questionable metascience practice
</em>
(QMP) is a research practice, assumption, or perspective that’s been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. I discuss 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science.

Questionable Metascience Practices

Researchers often distinguish between:

\(1\) Exploratory hypothesis tests --- unplanned tests of post hoc
hypotheses that may be based on the current results, and

\(2\) Confirmatory hypothesis tests --- planned tests of a priori
hypotheses that are independent from the current results

This distinction is supposed to be useful because exploratory results
are assumed to be more "tentative" and "open to bias" than confirmatory
results. In this [paper](https://doi.org/10.1080/09515089.2022.2113771),
we challenge this assumption and argue that exploratory results can be
***more*** compelling than confirmatory results.

Our article has three parts. In the first part, we demonstrate that the
same data can be used to generate and test a hypothesis in a
transparently valid manner. We agree that circular reasoning can
invalidate some exploratory hypothesis tests. However, circular
reasoning can be identified by checking the ***contents*** of the
reasoning without knowing the ***timing*** of that reasoning (i.e., a
priori or post hoc).

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*bzE3T0N_RJYeBZBUZPIDXw.png" />
<figcaption>Figure 1. An illustration of two ways in which exploratory
data analyses may provide legitimate support for post hoc
hypotheses</figcaption>
</figure>

In the second part of our article, we argue that exploratory hypothesis
tests can have several evidential ***advantages*** over confirmatory
tests and, consequently, they have the potential to deliver more
compelling research conclusions. In particular, exploratory hypothesis
tests:

✅ avoid researcher commitment and prophecy biases

✅ reduce the motive for data fraud

✅ are more appropriate following unplanned deviations

✅ facilitate inference to the best explanation

✅ allow peer reviewers to contribute to exploratory analyses

Finally, in the third part of our article, we consider several potential
\*disadvantages\* of exploratory hypothesis tests and conclude that
these potential disadvantages may not be problematic. In particular,
exploratory hypotheses tests are not necessarily disadvantaged due to:

✅overfitting

✅bias

✅HARKing

✅unacceptable research practices

And they:

✅are usually necessary

✅can be falsified

✅can predict anything but may suffer an evaluative cost in doing so

To be clear, our claim is not that exploratory hypothesis tests are
***always*** more compelling than confirmatory tests or even that they
are ***typically*** more compelling. Our claim is only that exploratory
tests ***can be*** more compelling in specific research situations. More
generally, we encourage researchers to evaluate specific tests and
results on a case-by-case basis rather than to follow simplistic
heuristics such as "exploratory results are more tentative," which
represents a form of methodolatory.

Our paper builds on some of my previous work on
[preregistration](https://doi.org/10.20982/tqmp.16.4.p376) and
[HARKing](https://drive.google.com/file/d/1bGIUjHSEAoJYJke6RWtBphXJjZLr1UeX/view).
And please check out Szollosi and Donkin's (2021)
[paper](https://doi.org/10.1177/1745691620966796) on "the misguided
distinction between exploratory and confirmatory research."

For more info, please see our open access article:

Rubin, M., & Donkin, C. (2022). Exploratory hypothesis tests can be more
compelling than confirmatory hypothesis tests. *Philosophical
Psychology*. <https://doi.org/10.1080/09515089.2022.2113771>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f55d6d52d09){width="1"
height="1"}


Researchers often distinguish between:  (1) Exploratory hypothesis tests — unplanned tests of post hoc hypotheses that may be based on the current results, and  (2) Confirmatory hypothesis tests — planned tests of a priori hypotheses that are independent from the current results  This distinction is supposed to be useful because exploratory results are assumed to be more “tentative” and “open to bias” than confirmatory results.

Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests

In this
[paper](https://www.google.com/url?q=https%3A%2F%2Fdoi.org%2F10.1037%2Fgpr0000135&sa=D&sntz=1&usg=AFQjCNGPXhRTKBRRFOs9Cvs9a3pW4QQRPQ)
(Rubin, 2017), I consider Gelman and Loken's (2013, 2014) garden of
forking paths problem. Forking paths occur when researchers decide which
analyses to perform based on information from their sample. For example,
a researcher may decide whether or not to drop an item from a scale
based on how it affects the scale's Cronbach alpha coefficient *within
the current sample.* This *sample-contingent decision rule* causes a
forking path in the analysis protocol because, during the course of
repeated sampling from the same population, some samples may yield an
acceptable Cronbach alpha value, leading to the retention of the item in
the scale, and some samples may yield an unacceptable Cronbach alpha,
leading to the deletion of the item.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*-41uiff_DcjfnK_X6y8JKQ.png" />
<figcaption>Source: <a
href="https://doi.org/10.1037/gpr0000135">https://doi.org/10.1037/gpr0000135</a></figcaption>
</figure>

Following the garden of forking paths would lead to multiple tests of
the same hypothesis during a long run of exact replications of the
analysis protocol (e.g., one test that included the item in the scale
and one test that excluded the item from the scale). Consequently, the
forking paths problem represents a special case of the multiple testing
problem. Forking paths increase the familywise Type I error rate for a
hypothesis by taking into account not only the test that was actually
conducted, but also the test that could have been conducted had a
different sample been drawn.

In my paper, I consider four potential solutions to the forking paths
problem: (a) preregistration, (b) sensitivity (robustness, multiverse)
analyses, (c) adjusting the alpha level, and (d) abandoning the
Neyman-Pearson approach. I conclude that preregistration and sensitivity
analyses are ineffective solutions, but that adjusting the alpha level
and/or abandoning the Neyman-Pearson approach are effective solutions.
In particular, the alpha level can be adjusted to take into account the
number of forking paths that are involved in testing the relevant
hypothesis (e.g., α/2 for one forking path, α/4 for two forking paths,
etc.). In addition, statistical inference approaches that do not rely on
the concept of repeated sampling from the same population are not
susceptible to the forking paths problem. Hence, the Fisherian and
Bayesian approaches to hypothesis testing do not suffer from the forking
paths problem because they both condition their probability statements
on the test that was actually conducted and the sample that was
actually drawn.

For further information, please see:

Rubin, M. (2017). An evaluation of four solutions to the forking paths
problem: Adjusted alpha, preregistration, sensitivity analyses, and
abandoning the Neyman-Pearson approach. *Review of General Psychology,
21*(4)*,* 321--329.
[https://doi.org/10.1037/gpr0000135](https://www.google.com/url?q=https%3A%2F%2Fdoi.org%2F10.1037%2Fgpr0000135&sa=D&sntz=1&usg=AFQjCNGPXhRTKBRRFOs9Cvs9a3pW4QQRPQ)
\*[Self-archived
version](https://drive.google.com/file/d/1wjOrFqqjw5aaISyKZANuRhXpjPkAPl1K/view?usp=sharing)\*

For more of my work in this area, please see:
<https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fd75a20f82d){width="1"
height="1"}


In this paper (Rubin, 2017), I consider Gelman and Loken’s (2013, 2014) garden of forking paths problem. Forking paths occur when researchers decide which analyses to perform based on information from their sample.

Rogue Scholar Posts

Questionable Metascience Practices

Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests

An Evaluation of Four Solutions to the Garden of Forking Paths Problem