Contained Experiments, Reproducibility Crisis?, Six Types of Metascience, and Personal Predictions

Contained Experiments

Over on Never Met a Science, discusses his new preprint on "Contained and Uncontained Experiments."

Contained experiments are those for which it is plausible to "perform the same procedure" and expect "the same result." Uncontained experiments are those for which practitioners expect intrinsic variation to dominate; even when performing the same procedure, the goal is to understand how results differ across time and context. I propose three conceptions of containedness: pragmatic (community expectations about replication), complexity (information required to specify the procedure), and temporal (speed of experimentation relative to drift in the phenomenon).

I was particularly interested in the pragmatic conception.

The pragmatic conception depends on the extent to which the conductors of an experiment and the scientific community with whom they interact expect the experiment to be able to be replicated. If a "contained" experiment is replicated and different outcome results, the community takes this as strong evidence for a problem. If the same occurs for an "uncontained" experiment, this is taken as evidence for some important variation in the context or implementation of the two experiments.

As Munger (2026) noted, Feest (2024) has made a related distinction between "effect seekers" and "context mongers." This distinction is important because it predicts

different flavors of solution to the replication crisis. The former are at least nominally interested in behavioralist stimulus-response patterns: these are the "effects" that they seek to produce and, then, re-produce when replicating a procedure. Their prescriptions tend to be methodological, premised on the idea that failures to re-produce these effects are caused by some combination of statistical and operational procedures. The "context mongers," in contrast, argue that this psychology research is undertheorized and that replication failures stem from an incomplete account of all of the dimensions of complexity involved in conducting those experiments.

Munger asks "why do different scientific communities have different pragmatic conceptions of containedness?" It's an important question that deserves more attention!

Reproducibility Crisis?

, Elizabeth Campolongo, and Phillip Popovich took part in a panel discussion at the Ohio State Center for Ethics and Human Values in which they responded to the question: "Is There a Reproducibility Crisis in Science?", focusing in particular on ethical and practical issues.

Popovich: "Let's face it: Science papers are stories, and the story rarely occurs in the order in which it is presented."

Devezer: On Bem's (2011) precognition study: "We know that is not a good study. Why run it again and again and again?"

Six Types of Metascience

and discuss "The Six Camps of Metascience" over on Macroscience.

If policymakers and agency leaders are going to change how they spend billions of dollars of taxpayer money based on metascience advice, they should have a clear picture of the landscape of metascience and its various goals, perspectives, and terms.

They identify six "camps" in metascience:

There have been other approaches to illustrating the diversity of metascience. In particular, Peterson and Panofsky (2023) located metascience at the intersection of the science of science, open science, and methodological activism.

My own, admittedly biased and incomplete, taxonomy would:

• characterize the promulgation of open science as "Activist Metascience" (Peterson & Panofsky, 2021);

• describe work investigating research integrity issues as "Forensic Metascience" (e.g., Heathers, 2025);

• merge Watney and Gustetic's categories of "innovation economics," "R&D management," and "institutional entrepreneurs" into a single category, perhaps called "Institutional Metascience"; and

• acknowledge four other lesser-known but nonetheless important areas:

  • "Theoretical Metascience" (e.g., Devezer, 2026; Munger, 2025)

  • "Feminist Metascience" (e.g., Cole & Gopalakrishna, 2025; Pownall, 2026)

  • "Critical Metascience" (e.g., Rubin, 2026; Ulpts & Bartscherer, 2025)

  • "Developmental Metaresearch" (e.g., Forscher & Schmidt, 2024; Schmidt, 2024)

So, I'd end up with the following…

"10 Years Ago…"

Mark Schaller published a paper on "the empirical benefits of conceptual rigor" in 2016 in which he argued that the replication crisis is partly due to researchers conceptualizing research hypotheses as personal predictions rather than theoretical predictions. Echoing Popper, he argued that:

Although scientists may subjectively experience hypotheses as their own personal predictions, a more formal approach to scientific inquiry treats scientific hypotheses as statements that have an independent logical status—independent of the scientists who articulate them and test them, and independent of the extent to which scientists personally believe them to be true or false.

Any truly rigorous approach to psychological science requires that scientific hypotheses cannot be equated to personal predictions; hypotheses must instead be articulated as depersonalized products of some systematic analysis, and appraised accordingly.

Schaller argued that conceiving hypotheses as personal predictions is likely to produce unrealistic expectations about replication rates because personal predictions are invested with self-serving biases and motivated reasoning that bias researchers towards viewing effects as less likely to be false positives and more likely to be larger and more generalizable than they really are. Hence:

If scientists can deliberately avoid thinking about hypotheses as personal predictions, then they are less likely to optimistically assume hypothesized effects to be truer, bigger, and broader than they actually are.

Subscribe now

Share