Mark Rubin

In this paper ([Rubin,
20](https://www.google.com/url?q=https%3A%2F%2Fdoi.org%2F10.1007%2Fs11229-019-02433-0&sa=D&sntz=1&usg=AFQjCNEVuCuq3elOVD71HY_Ee1DN_5q1Og)21),
I consider two types of Type I error probability. The *Neyman-Pearson
Type I error rate* refers to the maximum frequency of incorrectly
rejecting a null hypothesis if a test was to be repeatedly reconducted
on a series of different random samples that are all drawn from the
exact same null population. Hence, the Neyman-Pearson Type I error rate
refers to a long run of *exact* replications. In contrast, the
*Fisherian Type I error probability* is the probability of incorrectly
rejecting a null hypothesis in relation to a hypothetical population
that reflects the relevant characteristics of the particular sample
under consideration. Hence, the Fisherian Type I error rate refers to a
one-off sample rather than a series of samples that are drawn during a
long run of exact replications.

I argue that social science deals with units of analysis (people, social
groups, and social systems) that change over time. As the Greek
philosopher Heraclitus put it: "No man ever steps in the same river
twice, for it's not the same river and he's not the same man." Rivers
and men are what Schmidt (2009, p. 92) called "irreversible units" in
that they are complex time-sensitive systems that accumulate history.
The scientific investigation of these irreversible units cannot proceed
on the assumption that exact replications are possible. Consequently,
the Neyman-Pearson Type I error rate is irrelevant in social science,
because it relies on a concept of exact replication that cannot take
place in the case of people, social groups, and social systems. Why
should social scientists be interested in an error rate for an
impossible process of resampling from the same fixed and unchanging
population?

<figure>
<img
src="https://cdn-images-1.medium.com/max/554/1*zfDlYm8FEwiXhSQ8QrnI2g.jpeg" />
<figcaption>Source: <a
href="https://images.app.goo.gl/RiEP5tPX75AVbmFy6">https://images.app.goo.gl/RiEP5tPX75AVbmFy6</a></figcaption>
</figure>

I argue that the Fisherian Type I error probability is more appropriate
in social science because it refers to one-off samples from hypothetical
populations. In this case, researchers recognise that every sample comes
from a potentially different population. Hence, researchers can apply
the Fisherian Type I error probability to each sample-specific
provisional decision that they make about rejecting the same substantive
null hypothesis in a series of *direct* replications.

I conclude that the replication crisis may be partly (not wholly) due to
researchers' unrealistic expectations about replicability based on their
consideration of the Neyman-Pearson Type I error rate across a long run
of exact replications.

For further information, please see:

Rubin, M. (2021). What type of Type I error? Contrasting the
Neyman-Pearson and Fisherian approaches in the context of exact and
direct replications. *Synthese, 198,* 5809--5834*. *.
[https://doi.org/10.1007/s11229-019-02433-0](https://www.google.com/url?q=https%3A%2F%2Fdoi.org%2F10.1007%2Fs11229-019-02433-0&sa=D&sntz=1&usg=AFQjCNEVuCuq3elOVD71HY_Ee1DN_5q1Og)
\*[Self-archived
version](https://drive.google.com/file/d/10mEkuIElCGE5lYO-24HCYS8NKfytZBpG/view?usp=sharing)\*

I also discuss the differences between the Fisherian and Neyman-Pearson
approaches to hypothesis
testing [here](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/repeated-sampling).

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a32bac4af348){width="1"
height="1"}


In this paper (Rubin, 2021), I consider two types of Type I error probability. The
<em>
 Neyman-Pearson Type I error rate
</em>
refers to the maximum frequency of incorrectly rejecting a null hypothesis if a test was to be repeatedly reconducted on a series of different random samples that are all drawn from the exact same null population. Hence, the Neyman-Pearson Type I error rate refers to a long run of
<em>
 exact
</em>
replications.

What Type of Type I Error?

In this paper ([Rubin,
2020](https://drive.google.com/file/d/13cWuP8ilmQWMSORmzr6pLf3mJtn2C0jx/view?usp=sharing)),
I consider Fisher's criticism that the Neyman-Pearson approach to
hypothesis testing relies on the assumption of "repeated sampling from
the same population." This criticism is problematic for the
Neyman-Pearson approach because it implies that test users need to know,
for sure, what counts as the same or equivalent population as their
current population. If they don't know what counts as the same or
equivalent population, then they can't specify a procedure that would be
able to repeatedly sample from this population, rather than from other
non-equivalent populations, and without this specification the
Neyman-Pearson long run error rates become meaningless.

I argue that, by definition, researchers do not know for sure what are
the relevant and irrelevant features of their current populations. For
example, in a psychology study, is the population "1st year
undergraduate psychology students" or, more narrowly, "Australian 1st
year undergraduate psychology students" or, more broadly, "psychology
undergraduate students" or, even more broadly, "young people," etc.?
Researchers can make educated guesses about the relevant and irrelevant
aspects of their population. However, they must concede that those
guesses may be wrong. Consequently, if a researcher imagines a long run
of repeated sampling, then they must imagine that they would make
incorrect decisions about their null hypothesis due to not only Type I
errors and Type II errors, but also Type III errors --- errors caused by
accidentally sampling from populations that are substantively different
to their underspecified alternative and null populations.

To be clear, the Neyman-Pearson approach *does* consider Type III
errors. However, it considers them *outside* of each long run of
repeated sampling. It does not allow Type III errors to occur *inside* a
long run of repeated sampling, where the sampling must always be from a
correctly specified family of "admissible" populations (Neyman, 1977, p.
106; Neyman & Pearson, 1933, p. 294). In my paper, I argue that
researchers are unable to imagine a long run of repeated sampling from
the same or equivalent populations as their current population because
they are unclear about the relevant and irrelevant characteristics of
their current population. Consequently, they are unable to rule out Type
III errors within their imagined long run.

Following Fisher, I contrast scientific researchers with quality
controllers in industrial production settings. Unlike researchers,
quality controllers have clear knowledge about the relevant and
irrelevant characteristics of their populations. For example, they are
given a clear and unequivocal definition of Batch 57 on a production
line, and they don't consider re-conceptualizing Batch 57 as including
or excluding other features. They also know which aspects of their
testing procedure are relevant and irrelevant, and they are provided
with precise quality control standards that allow them to know, for
sure, their smallest effect size of interest. Consequently, the
Neyman-Pearson approach is suitable for quality controllers because
quality controllers can imagine a testing process that repeatedly draws
random samples from the same population over and over again. In
contrast, the Neyman-Pearson approach is not appropriate in scientific
investigations because researchers do not have a clear understanding of
the relevant and irrelevant aspects of their populations, their tests,
or the smallest effect size that represents their population. Indeed,
they are "researchers" because they are "researching" these things.
Hence, it is researchers' self-declared ignorance and doubt about the
nature of their populations that renders Neyman-Pearson long run error
rates scientifically meaningless.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*uUO4ovE7vJWkr5NObePNBw.jpeg" />
<figcaption>Source: <a
href="https://dissolve.com/stock-photo/Quality-control-workers-digital-tablet-examining-blocks-royalty-free-image/101-D1230-1-512">https://dissolve.com/stock-photo/Quality-control-workers-digital-tablet-examining-blocks-royalty-free-image/101-D1230-1-512</a></figcaption>
</figure>

For further information, please see:

Rubin, M. (2020). "Repeated sampling from the same population?" A
critique of Neyman and Pearson's responses to Fisher. *European Journal
for Philosophy of Science, 10*, Article 42, 1--15*.*
[https://doi.org/10.1007/s13194-020-00309-6](https://www.google.com/url?q=https%3A%2F%2Fdoi.org%2F10.1007%2Fs13194-020-00309-6&sa=D&sntz=1&usg=AFQjCNHY2NhGe0PmkpF4-n_IgbhMNd4z3A)
\*[Publisher's open access view only
version](https://www.google.com/url?q=https%3A%2F%2Frdcu.be%2Fb7K0s&sa=D&sntz=1&usg=AFQjCNGf6S0omDOBOhaS48krOpG8TgGDQA)\*
\*[Author's
version](https://drive.google.com/file/d/13cWuP8ilmQWMSORmzr6pLf3mJtn2C0jx/view?usp=sharing)\*

I also discuss the differences between the Fisherian and Neyman-Pearson
approaches to hypothesis
testing [here](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/what-is-a-type-i-error).

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=590ef01327d2){width="1"
height="1"}


In this paper (Rubin, 2020), I consider Fisher’s criticism that the Neyman-Pearson approach to hypothesis testing relies on the assumption of “repeated sampling from the same population.” This criticism is problematic for the Neyman-Pearson approach because it implies that test users need to know, for sure, what counts as the same or equivalent population as their current population.

Messages de Rogue Scholar

What Type of Type I Error?

Repeated Sampling from the Same Population?