Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests
Researchers often distinguish between:
(1) Exploratory hypothesis tests — unplanned tests of post hoc hypotheses that may be based on the current results, and
(2) Confirmatory hypothesis tests — planned tests of a priori hypotheses that are independent from the current results
This distinction is supposed to be useful because exploratory results are assumed to be more “tentative” and “open to bias” than confirmatory results. In this paper, we challenge this assumption and argue that exploratory results can be more compelling than confirmatory results.
Our article has three parts. In the first part, we demonstrate that the same data can be used to generate and test a hypothesis in a transparently valid manner. We agree that circular reasoning can invalidate some exploratory hypothesis tests. However, circular reasoning can be identified by checking the contents of the reasoning without knowing the timing of that reasoning (i.e., a priori or post hoc).
In the second part of our article, we argue that exploratory hypothesis tests can have several evidential advantages over confirmatory tests and, consequently, they have the potential to deliver more compelling research conclusions. In particular, exploratory hypothesis tests:
✅ avoid researcher commitment and prophecy biases
✅ reduce the motive for data fraud
✅ are more appropriate following unplanned deviations
✅ facilitate inference to the best explanation
✅ allow peer reviewers to contribute to exploratory analyses
Finally, in the third part of our article, we consider several potential *disadvantages* of exploratory hypothesis tests and conclude that these potential disadvantages may not be problematic. In particular, exploratory hypotheses tests are not necessarily disadvantaged due to:
✅overfitting
✅bias
✅HARKing
✅unacceptable research practices
And they:
✅are usually necessary
✅can be falsified
✅can predict anything but may suffer an evaluative cost in doing so
To be clear, our claim is not that exploratory hypothesis tests are always more compelling than confirmatory tests or even that they are typically more compelling. Our claim is only that exploratory tests can be more compelling in specific research situations. More generally, we encourage researchers to evaluate specific tests and results on a case-by-case basis rather than to follow simplistic heuristics such as “exploratory results are more tentative,” which represents a form of methodolatory.
Our paper builds on some of my previous work on preregistration and HARKing. And please check out Szollosi and Donkin’s (2021) paper on “the misguided distinction between exploratory and confirmatory research.”
For more info, please see our open access article:
Rubin, M., & Donkin, C. (2022). Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests. Philosophical Psychology. https://doi.org/10.1080/09515089.2022.2113771
