Mark Rubin

Researchers often distinguish between:

\(1\) Exploratory hypothesis tests --- unplanned tests of post hoc
hypotheses that may be based on the current results, and

\(2\) Confirmatory hypothesis tests --- planned tests of a priori
hypotheses that are independent from the current results

This distinction is supposed to be useful because exploratory results
are assumed to be more "tentative" and "open to bias" than confirmatory
results. In this [paper](https://doi.org/10.1080/09515089.2022.2113771),
we challenge this assumption and argue that exploratory results can be
***more*** compelling than confirmatory results.

Our article has three parts. In the first part, we demonstrate that the
same data can be used to generate and test a hypothesis in a
transparently valid manner. We agree that circular reasoning can
invalidate some exploratory hypothesis tests. However, circular
reasoning can be identified by checking the ***contents*** of the
reasoning without knowing the ***timing*** of that reasoning (i.e., a
priori or post hoc).

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*bzE3T0N_RJYeBZBUZPIDXw.png" />
<figcaption>Figure 1. An illustration of two ways in which exploratory
data analyses may provide legitimate support for post hoc
hypotheses</figcaption>
</figure>

In the second part of our article, we argue that exploratory hypothesis
tests can have several evidential ***advantages*** over confirmatory
tests and, consequently, they have the potential to deliver more
compelling research conclusions. In particular, exploratory hypothesis
tests:

✅ avoid researcher commitment and prophecy biases

✅ reduce the motive for data fraud

✅ are more appropriate following unplanned deviations

✅ facilitate inference to the best explanation

✅ allow peer reviewers to contribute to exploratory analyses

Finally, in the third part of our article, we consider several potential
\*disadvantages\* of exploratory hypothesis tests and conclude that
these potential disadvantages may not be problematic. In particular,
exploratory hypotheses tests are not necessarily disadvantaged due to:

✅overfitting

✅bias

✅HARKing

✅unacceptable research practices

And they:

✅are usually necessary

✅can be falsified

✅can predict anything but may suffer an evaluative cost in doing so

To be clear, our claim is not that exploratory hypothesis tests are
***always*** more compelling than confirmatory tests or even that they
are ***typically*** more compelling. Our claim is only that exploratory
tests ***can be*** more compelling in specific research situations. More
generally, we encourage researchers to evaluate specific tests and
results on a case-by-case basis rather than to follow simplistic
heuristics such as "exploratory results are more tentative," which
represents a form of methodolatory.

Our paper builds on some of my previous work on
[preregistration](https://doi.org/10.20982/tqmp.16.4.p376) and
[HARKing](https://drive.google.com/file/d/1bGIUjHSEAoJYJke6RWtBphXJjZLr1UeX/view).
And please check out Szollosi and Donkin's (2021)
[paper](https://doi.org/10.1177/1745691620966796) on "the misguided
distinction between exploratory and confirmatory research."

For more info, please see our open access article:

Rubin, M., & Donkin, C. (2022). Exploratory hypothesis tests can be more
compelling than confirmatory hypothesis tests. *Philosophical
Psychology*. <https://doi.org/10.1080/09515089.2022.2113771>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f55d6d52d09){width="1"
height="1"}


Researchers often distinguish between:  (1) Exploratory hypothesis tests — unplanned tests of post hoc hypotheses that may be based on the current results, and  (2) Confirmatory hypothesis tests — planned tests of a priori hypotheses that are independent from the current results  This distinction is supposed to be useful because exploratory results are assumed to be more “tentative” and “open to bias” than confirmatory results.

Messaggi di Rogue Scholar

Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests