Mark Rubin

In this paper ([Rubin,
2020](https://drive.google.com/file/d/13cWuP8ilmQWMSORmzr6pLf3mJtn2C0jx/view?usp=sharing)),
I consider Fisher's criticism that the Neyman-Pearson approach to
hypothesis testing relies on the assumption of "repeated sampling from
the same population." This criticism is problematic for the
Neyman-Pearson approach because it implies that test users need to know,
for sure, what counts as the same or equivalent population as their
current population. If they don't know what counts as the same or
equivalent population, then they can't specify a procedure that would be
able to repeatedly sample from this population, rather than from other
non-equivalent populations, and without this specification the
Neyman-Pearson long run error rates become meaningless.

I argue that, by definition, researchers do not know for sure what are
the relevant and irrelevant features of their current populations. For
example, in a psychology study, is the population "1st year
undergraduate psychology students" or, more narrowly, "Australian 1st
year undergraduate psychology students" or, more broadly, "psychology
undergraduate students" or, even more broadly, "young people," etc.?
Researchers can make educated guesses about the relevant and irrelevant
aspects of their population. However, they must concede that those
guesses may be wrong. Consequently, if a researcher imagines a long run
of repeated sampling, then they must imagine that they would make
incorrect decisions about their null hypothesis due to not only Type I
errors and Type II errors, but also Type III errors --- errors caused by
accidentally sampling from populations that are substantively different
to their underspecified alternative and null populations.

To be clear, the Neyman-Pearson approach *does* consider Type III
errors. However, it considers them *outside* of each long run of
repeated sampling. It does not allow Type III errors to occur *inside* a
long run of repeated sampling, where the sampling must always be from a
correctly specified family of "admissible" populations (Neyman, 1977, p.
106; Neyman & Pearson, 1933, p. 294). In my paper, I argue that
researchers are unable to imagine a long run of repeated sampling from
the same or equivalent populations as their current population because
they are unclear about the relevant and irrelevant characteristics of
their current population. Consequently, they are unable to rule out Type
III errors within their imagined long run.

Following Fisher, I contrast scientific researchers with quality
controllers in industrial production settings. Unlike researchers,
quality controllers have clear knowledge about the relevant and
irrelevant characteristics of their populations. For example, they are
given a clear and unequivocal definition of Batch 57 on a production
line, and they don't consider re-conceptualizing Batch 57 as including
or excluding other features. They also know which aspects of their
testing procedure are relevant and irrelevant, and they are provided
with precise quality control standards that allow them to know, for
sure, their smallest effect size of interest. Consequently, the
Neyman-Pearson approach is suitable for quality controllers because
quality controllers can imagine a testing process that repeatedly draws
random samples from the same population over and over again. In
contrast, the Neyman-Pearson approach is not appropriate in scientific
investigations because researchers do not have a clear understanding of
the relevant and irrelevant aspects of their populations, their tests,
or the smallest effect size that represents their population. Indeed,
they are "researchers" because they are "researching" these things.
Hence, it is researchers' self-declared ignorance and doubt about the
nature of their populations that renders Neyman-Pearson long run error
rates scientifically meaningless.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*uUO4ovE7vJWkr5NObePNBw.jpeg" />
<figcaption>Source: <a
href="https://dissolve.com/stock-photo/Quality-control-workers-digital-tablet-examining-blocks-royalty-free-image/101-D1230-1-512">https://dissolve.com/stock-photo/Quality-control-workers-digital-tablet-examining-blocks-royalty-free-image/101-D1230-1-512</a></figcaption>
</figure>

For further information, please see:

Rubin, M. (2020). "Repeated sampling from the same population?" A
critique of Neyman and Pearson's responses to Fisher. *European Journal
for Philosophy of Science, 10*, Article 42, 1--15*.*
[https://doi.org/10.1007/s13194-020-00309-6](https://www.google.com/url?q=https%3A%2F%2Fdoi.org%2F10.1007%2Fs13194-020-00309-6&sa=D&sntz=1&usg=AFQjCNHY2NhGe0PmkpF4-n_IgbhMNd4z3A)
\*[Publisher's open access view only
version](https://www.google.com/url?q=https%3A%2F%2Frdcu.be%2Fb7K0s&sa=D&sntz=1&usg=AFQjCNGf6S0omDOBOhaS48krOpG8TgGDQA)\*
\*[Author's
version](https://drive.google.com/file/d/13cWuP8ilmQWMSORmzr6pLf3mJtn2C0jx/view?usp=sharing)\*

I also discuss the differences between the Fisherian and Neyman-Pearson
approaches to hypothesis
testing [here](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/what-is-a-type-i-error).

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=590ef01327d2){width="1"
height="1"}


In this paper (Rubin, 2020), I consider Fisher’s criticism that the Neyman-Pearson approach to hypothesis testing relies on the assumption of “repeated sampling from the same population.” This criticism is problematic for the Neyman-Pearson approach because it implies that test users need to know, for sure, what counts as the same or equivalent population as their current population.

Repeated Sampling from the Same Population?

In this
[paper](https://www.google.com/url?q=https%3A%2F%2Fdoi.org%2F10.1037%2Fgpr0000135&sa=D&sntz=1&usg=AFQjCNGPXhRTKBRRFOs9Cvs9a3pW4QQRPQ)
(Rubin, 2017), I consider Gelman and Loken's (2013, 2014) garden of
forking paths problem. Forking paths occur when researchers decide which
analyses to perform based on information from their sample. For example,
a researcher may decide whether or not to drop an item from a scale
based on how it affects the scale's Cronbach alpha coefficient *within
the current sample.* This *sample-contingent decision rule* causes a
forking path in the analysis protocol because, during the course of
repeated sampling from the same population, some samples may yield an
acceptable Cronbach alpha value, leading to the retention of the item in
the scale, and some samples may yield an unacceptable Cronbach alpha,
leading to the deletion of the item.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*-41uiff_DcjfnK_X6y8JKQ.png" />
<figcaption>Source: <a
href="https://doi.org/10.1037/gpr0000135">https://doi.org/10.1037/gpr0000135</a></figcaption>
</figure>

Following the garden of forking paths would lead to multiple tests of
the same hypothesis during a long run of exact replications of the
analysis protocol (e.g., one test that included the item in the scale
and one test that excluded the item from the scale). Consequently, the
forking paths problem represents a special case of the multiple testing
problem. Forking paths increase the familywise Type I error rate for a
hypothesis by taking into account not only the test that was actually
conducted, but also the test that could have been conducted had a
different sample been drawn.

In my paper, I consider four potential solutions to the forking paths
problem: (a) preregistration, (b) sensitivity (robustness, multiverse)
analyses, (c) adjusting the alpha level, and (d) abandoning the
Neyman-Pearson approach. I conclude that preregistration and sensitivity
analyses are ineffective solutions, but that adjusting the alpha level
and/or abandoning the Neyman-Pearson approach are effective solutions.
In particular, the alpha level can be adjusted to take into account the
number of forking paths that are involved in testing the relevant
hypothesis (e.g., α/2 for one forking path, α/4 for two forking paths,
etc.). In addition, statistical inference approaches that do not rely on
the concept of repeated sampling from the same population are not
susceptible to the forking paths problem. Hence, the Fisherian and
Bayesian approaches to hypothesis testing do not suffer from the forking
paths problem because they both condition their probability statements
on the test that was actually conducted and the sample that was
actually drawn.

For further information, please see:

Rubin, M. (2017). An evaluation of four solutions to the forking paths
problem: Adjusted alpha, preregistration, sensitivity analyses, and
abandoning the Neyman-Pearson approach. *Review of General Psychology,
21*(4)*,* 321--329.
[https://doi.org/10.1037/gpr0000135](https://www.google.com/url?q=https%3A%2F%2Fdoi.org%2F10.1037%2Fgpr0000135&sa=D&sntz=1&usg=AFQjCNGPXhRTKBRRFOs9Cvs9a3pW4QQRPQ)
\*[Self-archived
version](https://drive.google.com/file/d/1wjOrFqqjw5aaISyKZANuRhXpjPkAPl1K/view?usp=sharing)\*

For more of my work in this area, please see:
<https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fd75a20f82d){width="1"
height="1"}


In this paper (Rubin, 2017), I consider Gelman and Loken’s (2013, 2014) garden of forking paths problem. Forking paths occur when researchers decide which analyses to perform based on information from their sample.

Messages de Rogue Scholar

Repeated Sampling from the Same Population?

An Evaluation of Four Solutions to the Garden of Forking Paths Problem