Mark Rubin

In this paper (Rubin, 2022), I make two related points: (1) researchers should halve two-sided

 p

values if they wish to use them to make directional claims, and (2) researchers should

 not

halve their alpha level if they’re using two one-sided tests to test two directional null hypotheses.

One-Sided Significance Tests

Researchers often distinguish between:  (1) Exploratory hypothesis tests — unplanned tests of post hoc hypotheses that may be based on the current results, and  (2) Confirmatory hypothesis tests — planned tests of a priori hypotheses that are independent from the current results  This distinction is supposed to be useful because exploratory results are assumed to be more “tentative” and “open to bias” than confirmatory results.

Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests

In this paper (Rubin, 2017), I consider Gelman and Loken’s (2013, 2014) garden of forking paths problem. Forking paths occur when researchers decide which analyses to perform based on information from their sample.

An Evaluation of Four Solutions to the Garden of Forking Paths Problem

The Costs of HARKing: Does it Matter if Researchers Engage in Undisclosed Hypothesizing After the Results are Known? While no-one’s looking, a Texas sharpshooter fires his gun at a barn wall, walks up to his bullet holes, and paints targets around them. When his friends arrive, he points at the targets and claims he’s a good shot.

The Costs of HARKing: Does it Matter if Researchers Engage in Undisclosed Hypothesizing After the…

In this new article, I consider questionable research practices in the field of metascience. A

 questionable metascience practice

(QMP) is a research practice, assumption, or perspective that’s been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. I discuss 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science.

Questionable Metascience Practices

Preregistration entails researchers registering their planned research hypotheses, methods, and analyses in a time-stamped document before they undertake their data collection and analyses. This document is then made available with the published research report in order to allow readers to identify discrepancies between what the researchers originally planned to do and what they actually ended up doing.

Does Preregistration Improve the Interpretablity and Credibility of Research Findings?

In this new paper (Rubin, 2021), I consider when researchers should adjust their alpha level (significance threshold) during multiple testing and multiple comparisons. I consider three types of multiple testing (disjunction, conjunction, and individual), and I argue that an alpha adjustment is only required for one of these three types.

When to Adjust Alpha During Multiple Testing

In this paper (Rubin, 2021), I consider two types of Type I error probability. The

 Neyman-Pearson Type I error rate

refers to the maximum frequency of incorrectly rejecting a null hypothesis if a test was to be repeatedly reconducted on a series of different random samples that are all drawn from the exact same null population. Hence, the Neyman-Pearson Type I error rate refers to a long run of

 exact

replications.

What Type of Type I Error?

In this paper (Rubin, 2020), I consider Fisher’s criticism that the Neyman-Pearson approach to hypothesis testing relies on the assumption of “repeated sampling from the same population.” This criticism is problematic for the Neyman-Pearson approach because it implies that test users need to know, for sure, what counts as the same or equivalent population as their current population.

Repeated Sampling from the Same Population?

Stories by Mark Rubin on Medium

In this
[paper](https://www.google.com/url?q=https%3A%2F%2Fdoi.org%2F10.1037%2Fgpr0000135&sa=D&sntz=1&usg=AFQjCNGPXhRTKBRRFOs9Cvs9a3pW4QQRPQ)
(Rubin, 2017), I consider Gelman and Loken's (2013, 2014) garden of
forking paths problem. Forking paths occur when researchers decide which
analyses to perform based on information from their sample. For example,
a researcher may decide whether or not to drop an item from a scale
based on how it affects the scale's Cronbach alpha coefficient *within
the current sample.* This *sample-contingent decision rule* causes a
forking path in the analysis protocol because, during the course of
repeated sampling from the same population, some samples may yield an
acceptable Cronbach alpha value, leading to the retention of the item in
the scale, and some samples may yield an unacceptable Cronbach alpha,
leading to the deletion of the item.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*-41uiff_DcjfnK_X6y8JKQ.png" />
<figcaption>Source: <a
href="https://doi.org/10.1037/gpr0000135">https://doi.org/10.1037/gpr0000135</a></figcaption>
</figure>

Following the garden of forking paths would lead to multiple tests of
the same hypothesis during a long run of exact replications of the
analysis protocol (e.g., one test that included the item in the scale
and one test that excluded the item from the scale). Consequently, the
forking paths problem represents a special case of the multiple testing
problem. Forking paths increase the familywise Type I error rate for a
hypothesis by taking into account not only the test that was actually
conducted, but also the test that could have been conducted had a
different sample been drawn.

In my paper, I consider four potential solutions to the forking paths
problem: (a) preregistration, (b) sensitivity (robustness, multiverse)
analyses, (c) adjusting the alpha level, and (d) abandoning the
Neyman-Pearson approach. I conclude that preregistration and sensitivity
analyses are ineffective solutions, but that adjusting the alpha level
and/or abandoning the Neyman-Pearson approach are effective solutions.
In particular, the alpha level can be adjusted to take into account the
number of forking paths that are involved in testing the relevant
hypothesis (e.g., α/2 for one forking path, α/4 for two forking paths,
etc.). In addition, statistical inference approaches that do not rely on
the concept of repeated sampling from the same population are not
susceptible to the forking paths problem. Hence, the Fisherian and
Bayesian approaches to hypothesis testing do not suffer from the forking
paths problem because they both condition their probability statements
on the test that was actually conducted and the sample that was
actually drawn.

For further information, please see:

Rubin, M. (2017). An evaluation of four solutions to the forking paths
problem: Adjusted alpha, preregistration, sensitivity analyses, and
abandoning the Neyman-Pearson approach. *Review of General Psychology,
21*(4)*,* 321--329.
[https://doi.org/10.1037/gpr0000135](https://www.google.com/url?q=https%3A%2F%2Fdoi.org%2F10.1037%2Fgpr0000135&sa=D&sntz=1&usg=AFQjCNGPXhRTKBRRFOs9Cvs9a3pW4QQRPQ)
\*[Self-archived
version](https://drive.google.com/file/d/1wjOrFqqjw5aaISyKZANuRhXpjPkAPl1K/view?usp=sharing)\*

For more of my work in this area, please see:
<https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fd75a20f82d){width="1"
height="1"}