The article “Resolving empirical controversies with mechanistic evidence” discusses the potential of using evidence about mechanisms to resolve statistical disagreements and aid in choosing the correct quantitative model. While there are challenges and uncertainties in this approach, it emphasizes the value of theorizing about mechanisms and collecting evidence about them, especially in disciplines like economics.

Ingo Rohlfing

Recently, I came across the open access article [Resolving empirical
controversies with mechanistic
evidence](https://link.springer.com/article/10.1007/s11229-021-03232-2).
I have some thoughts on the arguments made in the article, but first a
disclaimer: This is not meant to be a disciplinary beauty contest.

The article raises a valuable point: Two or more quantitative models may
produce contradictory results. Depending on the research question at
hand, it may be possible to use quantitative evidence, specification
tests etc. to choose one correct model (leaving aside here what
"correct" means). Let's say this is the stand-alone quantitative
approach toward adjudicating between models. A complementary strategy is
to use evidence about mechanisms that connect the cause to the outcome.

First, the points I agree with:  
1) I concur with the argument that evidence about mechanisms *can*
resolve statistical disagrements. This particularly concerns questions
of what variables to include in a model. Statistical controversies about
whether to estimate fixed effects or random effects are unlikely to be
resolvable through mechanism evidence because this is an estimation
matter.  
2) Economics, which is primarily quantitatively oriented and the focus
of the article, has much potential to benefit from incorporating
mechanism evidence *if* it is qualitative.

Now brief thoughts on the argument: From my peripheral knowledge of
economics, there is discussion of mechanisms, understood as mediators,
that are studied quantitatively. One can question whether mediators are
mechanisms; my point is that there are studies in economics, but also
[in political science](https://doi.org/10.1017/S0003055411000414), that
treat the two synonymously. As far as I understand evidential pluralism
as it was popularized by [Russo and
Williamson](https://doi.org/10.1080/02698590701498084), the analysis of
mediators would qualify as collecting evidence about mechanisms. (On a
slightly unrelated note, this still leaves me struggling with the
distinction between 'difference-making evidence' and 'mechanism
evidence' because [it seems
like](https://doi.org/10.1080/02698595.2019.1630927) the latter can also
be evidence that a mechanism makes a difference.)

The general literature about mixed methods, and the social science
literature about [nested
analysis](https://www.cambridge.org/core/journals/american-political-science-review/article/abs/nested-analysis-as-a-mixedmethod-strategy-for-comparative-research/D4FF59D175D761C20BD6CE440AFD700B)
in particular, have a long tradition of discussing strategies for
integrating different kinds of evidence collected and analyzed with
different methods. As the article on resolving empirical controversies
discusses at the end, evidence about mechanisms *may* help in settling
on one quantitative model, but not necessarily. The evidence about the
presence or absence of mechanisms may be too weak to make strong
inferences about whether and what mechanisms tie the cause to the
outcome. Or there may be good evidence for multiple possible mechanisms,
leaving one uncertain about what mechanism, or pair of mechanisms,
sustains a causal effect. Notwithstanding these issues that one may
confront, it is worth the effort to theorize mechanisms and to collect
evidence about them.


Recently, I came across the open access article Resolving empirical controversies with mechanistic evidence. I have some thoughts on the arguments made in the article, but first a disclaimer: This is not meant to be a disciplinary beauty contest. The article raises a valuable point: Two or more quantitative models may produce contradictory results.

“Resolving empirical controversies with mechanistic evidence” – Some thoughts

This post summarizes some (late) thoughts on the short article *[The
data revolution in social science needs qualitative
research](https://doi.org/10.1038/s41562-022-01333-7){rel="noreferrer noopener"
target="_blank"}* by Grigoropoulou and Small, published in Nature Human
Behavior. This is an excellent article that systemizes the ways in which
qualitative research should complement big data/computational social
science (CSS) and gives example of work that has done this already (I
understand big data/CSS to be the focus here).

From the perspective of political science, this complements calls for
combining qualitative and quantitative research

- in [nested
  analysis](https://doi.org/10.1017/S0003055405051762){rel="noreferrer noopener"
  target="_blank"} and in variants that have been developed later such
  as [pathway
  analysis](https://doi.org/10.1177/0049124114544420){rel="noreferrer noopener"
  target="_blank"}, or
- in [natural experiments](https://doi.org/10.1177/1065912907306470) and
  quasi-experimental designs to validate quantitative identification
  assumptions.

They, in turn, build at least implicitly on a much [longer history of
mixed-methods research](https://doi.org/10.1177%2F1558689815620883) in
other disciplines of the social sciences.

What I wish for is that the value of qualitative research is not
(re)discovered for every (new) quantitative method, but that it becomes
standard to understand that quantitative methods are based on at least
some assumptions that must be assessed with qualitative research. This
is not meant as criticism of the article in Nature Human Behavior
because it hopefully helps in contributing to such a general
understanding.

It is also important that one does qualitative *research* as a
complement to quantitative research that meets the rigor and standards
of whatever qualitative method is used. This might involve standards for
theory-centered case selection, evaluation of source credibility and the
balanced interpretation of evidence. The rising standards of qualitative
and quantitative methods suggest one best works in *mixed-methods
teams*. This point has been made by [George & Bennett in
2005](https://mitpress.mit.edu/books/case-studies-and-theory-development-social-sciences){target="_blank"
rel="noreferrer noopener"} already and is even more valid now than it
has been 17 years ago.


This post summarizes some (late) thoughts on the short article
<em>
 The data revolution in social science needs qualitative research
</em>
by Grigoropoulou and Small, published in Nature Human Behavior. This is an excellent article that systemizes the ways in which qualitative research should complement big data/computational social science (CSS) and gives example of work that has done this already (I understand big data/CSS to be the focus here).

Qualitative Methods in Computational Social Science

Review of the first chapter of the book “Multi-Method Social Science” by Jason Seawright, published with Cambridge University Press in 2016.

Over the next weeks, I will give a chapter-by-chapter review of the new
book [[Multi-Method Social Science: Combining Qualitative and
Quantitative
Tools](http://www.cambridge.org/de/academic/subjects/politics-international-relations/research-methods-politics/multi-method-social-science-combining-qualitative-and-quantitative-tools?format=PB){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} by [[Jason
Seawright](http://www.polisci.northwestern.edu/people/core-faculty/jason-seawright.html){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}. To my knowledge, it is the
first book on nested analysis/multimethod research in the social
sciences that is not an edited volume. A debate about mixed methods has
been [[going on in adjacent disciplines for
decades](http://dx.doi.org/10.1177/1558689815620883){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}, but the political science
development of nested analysis is largely running in parallel to the
mixed-methods debate (and vice versa). Because it is the first book and
there are many interesting and challenging arguments in it, I will
briefly review each chapter on its own terms. (Maybe I will discuss some
chapters in a single post; we will see because I have not finished
reading the book yet.)

In the opening chapter, Seawright distinguishes between *triangulation*
of qualitative and quantitative methods on the one hand, and the
*integration* of both methods on the other. He opposes the idea of
triangulating qualitative and quantitative methods for making a *single*
causal inference and answering the *same* research question because both
methods. In contrast, the integration of qualitative and quantitative
methods allows one to draw on their complementary strengths and to use
one method to check the assumptions underlying the other method (having
read chapter 3, it looks like this is a one-way street where the
qualitative part validates regression assumptions).

This sounds convincing, but it should not come as a big surprise for
those who are familiar with [[nested
analysis](https://doi.org/10.1017/S0003055405051762){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}. The key distinction
underlying nested analysis is one between a causal effect and a causal
mechanism. The inference about a causal effect is different from an
inference about a mechanism, meaning that triangulation as Seawright
defines it is notpossible. It is true that the term 'triangulation' is
used here and there in the multimethod literature, for example by
Lieberman in his seminal APSR article (p. 440). However, it is not meant
in the way Seawright defines triangulation. The idea of [[triangulation
plays a bigger role in the mixed-methods
literature](http://dx.doi.org/10.1177/1558689812437187){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}, which is, as I said, largely
detached from the nested analysis literature. We do not know, but I
assume that researchers who might feel challenged by Seawright's main
claim are not among his readership, and that the political science
readers of *Multi-Method Social Science* need not be convinced that
integration is central to nested analysis.

Substantively, I think the issue is that the key selling point of
multimethod research has been made in Lieberman's pathbreaking article.
This is not to say that there is no room for methodological work on
nested analysis because some elements still have to be addressed and
refined. However, the one reason why integrated research is valuable,
the combined analysis of causal effects and mechanisms, can only be made
once and has been made already. I should add that the formulation of a
punchline is probably more difficult and less needed for methods books
than substantive books because the value of methods books lies in taking
one through the details of a method.

Stylistically, it is not obvious who is targeted with the rejection of
triangulated causal inference because there is no recent reference cited
in this context (1995 is the most reference that is given). This
reinforces my impression that the central claim is endorsed by everyone
because nobody in the nested analysis community argues for triangulation
and against integration. More generally, my reading of chapter 1 (and
the following chapters) is that it is a little bit too light on the
referencing side. In some cases, a reference jumps to mind that is not
cited, including Lieberman's article at several places, and in some
cases the literature is misrepresented (like the chain of references
supposedly saying that multimethod research is not superior to
single-method research, p. 2). I find this unfortunate because it limits
the value of the first chapter for empirical researchers who want to use
it as a guide to existing discussions of multimethod research and
further readings (the chapters do not have a "Further Readings" section
at the end).

Stay tuned for chapter 2 on the causal foundations of multimethod
analysis.


Over the next weeks, I will give a chapter-by-chapter review of the new book Multi-Method Social Science: Combining Qualitative and Quantitative Tools by Jason Seawright. To my knowledge, it is the first book on nested analysis/multimethod research in the social sciences that is not an edited volume.

Messaggi di Rogue Scholar

“Resolving empirical controversies with mechanistic evidence” – Some thoughts

Qualitative Methods in Computational Social Science

Review of “Multi-Method Social Science” (Seawright, CUP): Chapter 1