The debate between Yann LeCun and Elon Musk, reported in Nature, questions whether science necessitates publishing results. My position is that science depends on how you produce your knowledge, not necessarily requiring publication. Only if you face the public, you need to publish about your work.

Ingo Rohlfing

Nature reports about a debate on X that started between Yann LeCun and Elon Musk. In short, the question is whether you can only call ‘science’ science when you publish your results. The Nature article is paywalled, but you can, as of now, still read the original post by LeCun that initiated the discussion.

Is it science when you don’t publish it?

Is Open Science passé?

is the question asked by Xenia Schmalz in this blogpost. I recommend reading it before I share brief thoughts on some points that are raised. I wish an open science movement was not needed anymore, but I agree this is most likely not the answer to the leading question. Neither has the open science movement failed; progress toward more transparent and credible science is simply slow.

Open Science is neither passé nor is it there yet

The LSE Impact blog has a post from May 2021 raising some reservations about the idea of ‘Slow Science’. The ‘Slow Science’ idea hasn’t really picked up in academia, as far as I can tell. The post presents some good thoughts about why the “slowness-idea” is problematic in general. I agree that slowness is not a value in itself. Sometimes, developments and events like a pandemic demand it to do research faster than one would do it otherwise.

Science should be as fast as possible and as slow as necessary to do good research

This post summarizes some (late) thoughts on the short article

 The data revolution in social science needs qualitative research

by Grigoropoulou and Small, published in Nature Human Behavior. This is an excellent article that systemizes the ways in which qualitative research should complement big data/computational social science (CSS) and gives example of work that has done this already (I understand big data/CSS to be the focus here).

Qualitative Methods in Computational Social Science

Some ideas about how the peer review process in academic publishing can be used as a device for preregistering parts of an empirical analysis.

Preregistration of an analysis requires the prior unobservability of data. One publicly declares what one intends to do before one can access data. One goal of preregistration is to diffuse potential concerns that a publication misrepresents the order in which an empirical analysis was implemented.

The peer review process as a preregistration device

Why the filling of gaps in the research literature is similar to replication and probably better than its reputation.

A common justification of empirical research is that a gap in the literature needs to be filled.

Gap-filling, puzzle-solving and replication in empirical research

On September 27, the German Science Foundation (DFG) announced its decision to award the status of a research cluster of excellence (Exzellenzcluster) to 57 cluster proposals from all disciplines.

Correlation vs causation: The case of competitive funding and research quality

About two weeks ago, COMPASSS issued a Statement on Rejecting Article Submissions because of QCA Solution Type. In short, the reasoning was that methodological work on QCA is developing and that reviewers and editors should not judge empirical work based on whether one particular solution type is interpreted as causal.

The COMPASSS statement and QCA solution types

The charge that Political Science (or other non-STEM disciplines) is lacking relevance and does not produce interesting research is made then and again, with two new pieces published these days. One is written by a political economist, stating that most research is boring;

The relevance of Political Science: Some thoughts on the recent critique

For what it’s worth, I wanted to see how often ‘process tracing’ is mentioned in the offerings on Google Books.

The rise of process tracing (in Google Books)

For some time now, a discussion has been raging about the pros and cons of set theory and the use of set-theoretic methods (STM) in the social sciences (e.g., in Sociological Methodology and the APSA Newsletter). Following up on a critical discussion by Paine and a constructive, comparative discussion of STM and regression analysis by Thiem, Baumgartner and Bol (TBB),

 Comparative Political Studies

organized a symposium on STM.

Another mistaken criticism of set theory and set-theoretic methods

Recently, I came across an interesting study from 2009, the content of which is not relevant here.

Two proposals for incentivizing data archiving

Let’s imagine you are attending a presentation and the presenter does not follow the standard intro-literature-theory-design-results-conclusion structure. Instead, she only presents the results of her research. What would be the first question you would ask? I assume at least 90 percent of the audience would ask about how she arrived at her findings.

Different tools, shared standards: The debate about DA-RT

At the end of last week, a two-day conference,

 Qualitative Comparative Analysis – Social Science Applications and Methodological Challenges

, took place in Tilburg, the Netherlands. Needless to say, the recent and ongoing wave of criticism of QCA was a key topic on the agenda and in discussions among participants.

QCA is developing… apart?

Open access and gated publishing have been discussed at length by researchers, publishers, and the media. Still, the debate got more intense when Elsevier sent takedown notices to Academia.edu. Yesterday, it got even more intense when it became public that the University of Calgary is also confronted with takedown notices from Elsevier. Elsevier responded to the critiques, but it is unlikely that this will take the heat out of the debate.

You may not like what Elsevier is doing, but…

Schlimm? 470000 entscheiden über die Koalition

Der Mitgliederentscheid wurde kritisiert, weil etwa 470000 SPD-Mitglieder über die große Koalition entscheiden.

Mit dem Quatsch aufhören III: Parteien entscheiden in eigener Sache

Der Mitgliederentscheid der SPD über den Koalitionsvertrag ist angelaufen. Je mehr Zeit seit der Ankündigung verstrichen ist, umso stärker und wilder scheint die Kritik zu werden, so dass ich nun doch einen Blogeintrag dazu schreibe (was eh wieder Zeit wurde). Aufgrund der Anhäufung von Kritik werde ich die über die nächsten Tage mehrere Einträge daraus machen.

 Alle Staatsgewalt geht vom Volke aus.

Mit dem Quatsch aufhören: Über die andauernde Kritik am SPD-Mitgliederentscheid und der Großen Koalition

Under the rubric of “Reader’s view”, I will post short reviews of books, mostly from the field of social science methods.

Reader’s view: Goertz/Mahoney (2012): A Tale of Two Cultures

The bi-annual publication of the APSA Section on Qualitative Methods and Multi-Method Research has a highly interesting and controversial symposium on confirmation bias in process tracing in its current issue.

Confirmation bias in causal qualitative research

How to properly interpret consistency and coverage values in fuzzy-set QCA.

In 2000, Ragin added the idea of fuzzy sets to Qualitative Comparative Analysis (QCA) and they have been widely used since then. However, one sometimes still finds misunderstandings about what fuzzy sets are, in particular when it comes to the interpretation of consistency and coverage scores. In a criticism of QCA by Stockemer (which is criticized for a variety of reasons and to which Stockemer again replied), he runs a fuzzy-set analysis.

Don’t conflate fuzzy set membership with cases in QCA

Alt-Bundespräsident Roman Herzog wird von Focus mit den Worten zitiert: „Im Prinzip ist die Fünf-Prozent-Hürde nicht mehr zeitgemäß. Eigentlich müssten wir die Hürde nach oben setzen.“ Die Begründung: Vor dem Hintergrund von immer mehr kleineren Parteien werde der Bundeskanzler/die Bundeskanzlerin ansonsten „nicht mehr von einer großen Mehrheit der Bevölkerung getragen“. Dies sieht Herzog als eine Gefährdung der parlamentarischen Demokratie.

Roman Herzog und die Hürde

Research on causal mechanisms is a growth industry, with the largest percentage of studies falling into the camp of qualitative research. The reason for this is the admonition that correlation is not causation, implying the claim that valid causal inference requires evidence on causal mechanisms.

Mechanisms in qualitative and quantitative research

In

 Finding Pathways: Mixed-Method Research for Studying Causal Mechanisms

, Weller and Barnes seek to explain “how the small-N component of multi-method research can meaningfully contribute and add value to the study of causal mechanisms” (quote from blurb). The book contains nine chapters, including the introduction and concluding chapter.

Review of “Finding Pathways”, by Weller/Barnes

Live and let live, or: unity and diversity in the social sciences

As an approach to and method for the analysis of set relations,

 Qualitative Comparative Analysis

(QCA) belongs in the toolbox of many social scientists (and organizational and management researchers alike). Although it has proven to be very popular, a question that one still often confronts is: what is QCA? Even those who understand the technicalities of QCA wonder what the qualitative part in QCA represents in practice.

What is QCA? Simple question, many answers

As is well known, QCA has been under intense scrutiny in recent years and subject to criticism (sometimes quite strong). I am not going to review studies on the validity of QCA that entail criticism;

Surprised, anyone? Putting the debate about QCA into context

Qualitative Comparative Analysis (QCA) is a method utilized by different disciplines in the social sciences and beyond, e.g., business economics and management. However, QCA users must still justify their choice of method more frequently than the users of other methods. Whatever the reason, it is actually not a bad thing to reflect upon the choice of a method because it should be suitable for answering our research question.

How (not) to justify running a QCA

If this was a blog post about

 the

#APSA2014, I would have to write about Friday night’s fire emergency at the Marriott (i.e., #APSAonfire) as the non-academic event that left a definite imprint (and affected me as one of the many people who had a room at the Marriott). But uncomfortable as that night had been, I will focus instead on the ongoing debate about set theory and Qualitative Comparative Analysis (QCA) which are

 the

Impressions from the APSA 2014, or: QCA under fire?

People familiar with the development of qualitative methods know that process tracing has developed rapidly over the last years. As the discussion about a method becomes broader and deeper, it becomes more important to systematize and sort the field in order to understand what it can and cannot achieve.

Three varieties of process tracing? It’s four

The choice of cases for empirical analysis is a central topic in the methods literature. The argument that quantitative and qualitative research studies different templates, which is probably most forcefully described in

 A Tale of Two Cultures

, does not come as a surprise. What I want to focus on here are the arguments on case selection in qualitative research.

Theory and substance in case selection, or: Give random selection a chance

The methods literature on process tracing took a Bayesian turn in recent years. Bayesian inference, whereby you condition on evidence in order to update your prior confidence in a hypothesis, is presented as a mode of inference one should follow in process tracing (Bennett, Andrew (2008): Process-Tracing: A Bayesian Perspective. Box-Steffensmeier, Janet M., Henry Brady and David Collier (ed.):

Frequentist inference and process tracing: an oxymoron?

Bekanntermaßen wurde der SPD-Mitgliederentscheid von einigen Verfassungsrechtlern in Frage gestellt. Mittlerweile hat das Verfassungsgericht in seiner Ablehnung eines Eilantrags auch deutlich gemacht, dass der Entscheid verfassungsrechtlich nicht zu beanstanden ist. Sigmar Gabriel hat die Verfassungsfrage des Mitgliederentscheid forsch beiseite gewischt, allerdings ist die Begründung irritierend (zumindest die, die er bei 3 nach 9 gegeben hat).

Die Verfassung und der Mitgliederentscheid: Wo Sigmar Gabriel irrt

Qualitative Methods (i.e., process tracing, set theoretic methods, informal Bayesian inference etc.) and multi-method research, in particular the combination of regression analysis or QCA with case studies, are certainly a growth industry in political science and sociology. In light of some methods panels held at the APSA Annual Meeting in Chicago and the ECPR General Conference in Bordeaux, this might actually come as a surprise.

Qualitative Methods: A Field in Search of Common Ground

Recently, Frontal 21 – a German newscast – reported about the handling of error in treatments under German health care law (about which I am not an expert). A lawyer specializing in health care law explained that it is very difficult to win lawsuits against medical doctors (MD’s) because the patient claiming to experience treatment error has to substantiate this claim (except for grave treatment error). Particularly problematic are cases where

Error in medical treatment and causation in German health care law

The current APSA Comparative Politics Newsletter is dedicated to “Doing Comparative Politics Elsewhere” (i.e. , outside of the US). Thomas Plümper contributes a discussion on Comparative Politics in Europe. In brief, Plümper argues that, until recently, the field of Comparative Politics (CP) has been dominated by qualitative methods.

Comparative politics and the choice of methods

Sourcely, an AI company, promises to streamline research by finding, summarizing, and adding credible sources in minutes. While this sounds appealing, skepticism arises as using such a tool may prioritize citing over genuine research. Initial tests revealed limited functionality, leaving doubts about its practical value in the research process.

I came across the AI company Sourcely these days. It promises to “

 Finish Your Research in Minutes. Save Your Sleep. Paste your essay to find, summarize, and add credible sources. (That’s something Google Scholar can’t do!)

” Saving time for sleep is good and if it would work, it would outperform GS, or any traditional search for citations and references, for that matter. However, I have some thoughts and concerns.

Citation-infusion of research papers with AI?

The article “Resolving empirical controversies with mechanistic evidence” discusses the potential of using evidence about mechanisms to resolve statistical disagreements and aid in choosing the correct quantitative model. While there are challenges and uncertainties in this approach, it emphasizes the value of theorizing about mechanisms and collecting evidence about them, especially in disciplines like economics.

Recently, I came across the open access article Resolving empirical controversies with mechanistic evidence. I have some thoughts on the arguments made in the article, but first a disclaimer: This is not meant to be a disciplinary beauty contest. The article raises a valuable point: Two or more quantitative models may produce contradictory results.

“Resolving empirical controversies with mechanistic evidence” – Some thoughts

In place of a generic blog post, I am reposting a short Twitter thread here. The thread is a response to an opinion piece on the Times Higher Education website titled Pay researchers for results, not plans. (Posts on the THE website require registration of an account that includes a couple of free reads.)  I copy-paste thread into this post. If you prefer to read it on Threadreader, you find it here.

Is payment for publications and paying for citations a better to way to fund research? Most likely not

When we use the Quine-McCluskey algorithm to derive a QCA solution, we can choose between the conservative, intermediate or parsimonious solution. While I do not have any figures about which solution has been produced how frequently in empirical research, it is safe to say that the conservative solution is quite popular.

The conservative QCA solution isn’t based on counterfactuals? Not so fast

When making causal (or descriptive) inferences, it is important to think about the context within which the causal relationship is expected to hold because it probably does not hold universally and, possibly, only in a limited setting. Falleti and Lynch have written an excellent article about “context” (I drop the ““ now) in relation with causal mechanisms.

“Context” is important, but (almost) useless if used as a causal category

One of the recent big and, in my view, underappreciated innovations in the field of Qualitative Comparative Analysis (QCA) is Baumgartner’s formulation of the Coincidence Analysis algorithm (CNA). Baumgartner presents it as an alternative to QCA, which I do not find convincing because I do not see QCA married to a specific algorithm.

You are a regularity theorist when using the Coincidence Analysis algorithm in QCA

More often than one might expect, television series and films offer excellent illustrations of methodological and methods-related arguments (which is worth a blog post of its own). When I was working on my paper on comparative hypothesis testing in process tracing, I was watching the first season of the terrific TV series, Homeland.

What Homeland has to do with causal inference in process tracing

Continuing the chapter-by-chapter review of Seawright’s book on Multi-Method Social Science took me longer than I imagined and it should have, but here we go again.

Review of ‘Multi-Method Social Science’ (Seawright, CUP) –  Chapter 2: Causation as A Shared Standard

Review of the first chapter of the book “Multi-Method Social Science” by Jason Seawright, published with Cambridge University Press in 2016.

Over the next weeks, I will give a chapter-by-chapter review of the new book Multi-Method Social Science: Combining Qualitative and Quantitative Tools by Jason Seawright. To my knowledge, it is the first book on nested analysis/multimethod research in the social sciences that is not an edited volume.

Review of “Multi-Method Social Science” (Seawright, CUP): Chapter 1

Das 2014er Gutachten der Expertenkommission Forschung und Innovation (EFI) schlug diese Woche einige Wellen, weil es nach wie vor eine Abwanderung der besten Wissenschaftler aus Deutschland konstatiert (

 brain drain

, wenn man den Begriff mag). Nach einem Blick in das Gutachten (Abschnitt B2, ab S. 85) sind dazu einige Anmerkungen angebracht, die die Schlagzeilen in den Kontext rücken. Erstens, die Ergebnisse sind wenig überraschend.

Das EFI-Gutachten und die Ab- und Zuwanderung deutscher Wissenschaftler: Ein paar kritische Anmerkungen

Inspired by an email exchange I had with someone on theoretically non-exclusive hypotheses in Bayesian process tracing, I believed it might be useful to write down some thoughts in a blog post. It ended up as a PDF on Github because it has a minor R element and formal notation.

Theoretically non-exclusive hypotheses in Bayesian process tracing

Many APSA 2016 panels and discussions in the Section on Qualitative and Multimethod Research and the Political Methodology Section were centered on the Data Access and Research Transparency (DART) Initiative (probably worth a blog post of its own). Even panels not explicitly dedicated to DART have digressed into this topic, which includes a short exchange between Tasha Fairfield and me in a panel on causal inference in qualitative research.

People require preregistration, not methods

Qualitative Comparative Analysis (QCA) is the method of choice for the analysis of set relations and has changed considerably and improved over the years.

Curiosities of QCA: Fuzzy-set consistency

The idea of most-likely and least-likely cases dates back to Eckstein and was one of the few remaining things in qualitative research there seemed to be no disagreement about because they are considered an asset in causal analysis. In a paper that is advance access, Beach and Pedersen (BP) now argue that process tracing and the analysis of mechanisms does not make sense with most-likely and least-likely cases.

Process tracing is possible with most-likely and least-likely cases

Among the recent criticisms of QCA, Lucas and Szatrowski’s (LS) critique stands out in multiple respects, including its scope, its tone (QCA is “a nonanalytic means to identify asymmetric causal illusions”, p. 66), and the responses it provoked (and the notorious mistake in their first simulation). Naturally, the critical replies focused on LS’s assessment of QCA, but I find that they make an almost equally interesting and ironic claim at the

The irony of Lucas/Szatrowski’s QCA criticism

How to enhance reproducibility when using fs/QCA software

The ability to reproduce published empirical studies is central to progress in science. Reproducibility requires the proper storage and sharing of data, which cannot be taken for granted, and a detailed, step-by-step description of the empirical analysis. Statistical research has no problems with the latter, in principle, because a script can be compiled (do-file etc.) that can be easily shared.

Reproducible QCA studies with fs/QCA

The social science literature is full of discussions about causation and what the best method for causal inference might be. However, a relatively small percentage of them draw on the philosophical debate about causation. Certainly, there is a great deal of talk about philosophy of science on the ontological and metaphysical level, including, for example, engagement with the relation between neo-positivism and realism.

Review of “Causation: A Very Short Introduction” by Mumford/Anjum

Most multi-method research (MMR) studies with which I am familiar start with regression analysis (or, in recent years, QCA) and perform the case studies afterward. This is the order recommended by Lieberman in his nested analysis article which, in turn, most probably reflects the widely held view that case studies are only worth doing in MMR when something is going on at the cross-case level.

What do we learn about case studies from follow-up regression analysis?

In einem Kommentar auf SPIEGEL ONLINE wird unter anderem der Mitgliederentscheid dafür verantwortlich gemacht, dass der Koalitionsvertrag wenig zukunftsorientiert sei und das Durchsetzen von Reformen zu Beginn der Legislaturperiode verhindere. Führt die Ausübung der innerparteilichen Demokratie also zu einer mutlosen Politik im hier und jetzt? Aller Wahrscheinlichkeit nach nicht.

Verhindert der Mitgliederentscheid zukunftsorientierte Politik?

Das Zögern und Zaudern der SPD vor, während und nach den Koalitionsverhandlungen wird gerne beiseite gewischt mit dem Hinweis, „der Wähler“ (und die Wählerin) wolle die große Koalition.

Mit dem Quatsch aufhören (II): “Der Wähler” will die Große Koalition

At the end of last week, I came across a blog entry by Olaf Storbeck, reporting that a rising star from German business economics, Prof. Ulrich Lichtenthaler, is faced with numerous inquiries concerning his publishing record. Two journals have already retracted three of his articles and additional articles are under scrutiny.

Quality and quantity in publishing

In the case study literature, one new pair of terms has been established by the field that concerns the distinction between

 data set observations

and

 causal process observations

. In short, Collier, Brady, and Seawright (2004: Sources of Leverage in Causal Inference: Toward an Alternative View of Methodology. Brady, Henry E. and David Collier (eds.):

 Rethinking Social Inquiry.

Don’t call it causal when it is an observation

Politics, Science, Political Science

Continuing the chapter-by-chapter review of [[Seawright's book on
Multi-Method Social
Science](http://www.cambridge.org/de/academic/subjects/politics-international-relations/research-methods-politics/multi-method-social-science-combining-qualitative-and-quantitative-tools?format=PB){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} took me longer than I imagined
and it should have, but here we go again. The second chapter discusses
the fundamentals of multimethod research (MMR) and identifies "Causation
as a Shared Standard". One might think that the fundamentals can be
quickly addressed because of the now established distinction between
causal effects, studied with a cross-case method, and causal mechanisms
that are studied with process tracing. However, this is not the route
the chapter is taking because it is mostly concerned with discussing the
[[potential
outcomes](http://economics.mit.edu/files/32){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} (PO) framework as the basis
for MMR.

In short, the PO framework couches causal analysis in a counterfactual
framework by asking what the outcome value of a unit would be if it had
been exposed to a different treatment. The first section of chapter 2
introduces the reader to the PO framework and explains how it relates to
different views on causation in the social sciences. [[Brady's seminal
discussion of
causation](http://dx.doi.org/10.1093/oxfordhb/9780199286546.003.0010){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} is the point of reference here
and leads Seawright to the conclusion that the PO framework is suitable
for integrating the existing perspectives -- regularity, manipulation,
mechanisms -- more or less.

If MMR involves a regression analysis, it is straightforward to rely on
the PO framework as the foundation of modern quantitative research.
However, my reading is that the first section treats other perspectives
on causation too lightly and tends to misrepresent their
complementarity. If one conceives of process tracing as a method that
allows one to make causal inferences with a single case (I don't, but
[[others
do](https://www.amazon.com/Tale-Two-Cultures-Qualitative-Quantitative/dp/0691149712/ref=sr_1_1?s=books&ie=UTF8&qid=1484777906&sr=1-1&keywords=a+tale+of+two+cultures){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}), it cannot be integrated into
a PO framework requiring counterfactual reasoning. Seawright's argument
that process tracing and the PO framework allow one to target
heterogeneity seems to force the issue a bit because these are different
types of heterogeneity.

At least of equal importance, I am hesitant to put the PO framework on
par with theories of causation because it is, above all, a framework for
the estimation of treatment effects of all sorts (so I read). I would
have loved to see a discussion of [[Pearl's directed acyclic graphs and
his methodological theory of causation using structural
equations](https://www.amazon.com/Causality-Reasoning-Inference-Judea-Pearl/dp/052189560X/ref=sr_1_1?s=books&ie=UTF8&qid=1484777986&sr=1-1&keywords=pearl+causality){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} to translate theoretical
expectations into causal models. Apart of putting theorizing center
stage, it might create more opportunities for linking regression
analysis and process tracing in MMR because there are also attempts to
[[formalize causal inference for single cases with structural
equations](https://www.amazon.com/Actual-Causality-Press-Joseph-Halpern/dp/0262035022/ref=sr_1_1?s=books&ie=UTF8&qid=1484778055&sr=1-1&keywords=actual+causality){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}. This is not to say that the
integration of both methods would be easy because, after all, they are
operating on different levels. Yet, if the goal is to formulate a
unified framework for MMR using similar terminology, this seems more
promising to me than the PO framework.

Chapter 2 then moves on with a short discussion of using regression
analysis for causal inference and a longer elaboration on regression
analysis as seen through the lenses of the PO framework. The sections
are written in more or less technical terms and might be new to
qualitative researchers and read as familiar for quantitative
researchers. In total, this means that regression analysis is central
throughout the entire chapter and foreshadows what becomes clearer when
reading the following chapters: For Seawright, the causal work is
entirely done by the quantitative part and the qualitative analysis
supports the quantitative inference in one way or the other (see the
forthcoming blog post of chapter 3).

Seawright's conception of MMR therefore differs considerably from what
one might believe to be the widely held idea that assigns equal weight
to the quantitative analysis (or QCA) and the process tracing part.
There is nothing wrong with this *per se*, but it might have been
appropriate to flag this upfront in the introduction. [[As I
explained](https://ingorohlfing.wordpress.com/2016/11/06/review-of-multi-method-social-science-seawright-cup-chapter-1/){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}, the key argument in the
introduction should hardly be contested in the political science
community on nested analysis, whereas the dominance of the quantitative
part in MMR might be. In this context, it is noteworthy that 'process
tracing' is not mentioned once in chapter 2 and causal mechanisms are
barely mentioned. It would have been interesting to learn in more detail
why Seawright's conception of MMR departs from nested analysis as
introduced by Lieberman, but this question remains unanswered.


Over the next weeks, I will give a chapter-by-chapter review of the new
book [[Multi-Method Social Science: Combining Qualitative and
Quantitative
Tools](http://www.cambridge.org/de/academic/subjects/politics-international-relations/research-methods-politics/multi-method-social-science-combining-qualitative-and-quantitative-tools?format=PB){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} by [[Jason
Seawright](http://www.polisci.northwestern.edu/people/core-faculty/jason-seawright.html){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}. To my knowledge, it is the
first book on nested analysis/multimethod research in the social
sciences that is not an edited volume. A debate about mixed methods has
been [[going on in adjacent disciplines for
decades](http://dx.doi.org/10.1177/1558689815620883){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}, but the political science
development of nested analysis is largely running in parallel to the
mixed-methods debate (and vice versa). Because it is the first book and
there are many interesting and challenging arguments in it, I will
briefly review each chapter on its own terms. (Maybe I will discuss some
chapters in a single post; we will see because I have not finished
reading the book yet.)

In the opening chapter, Seawright distinguishes between *triangulation*
of qualitative and quantitative methods on the one hand, and the
*integration* of both methods on the other. He opposes the idea of
triangulating qualitative and quantitative methods for making a *single*
causal inference and answering the *same* research question because both
methods. In contrast, the integration of qualitative and quantitative
methods allows one to draw on their complementary strengths and to use
one method to check the assumptions underlying the other method (having
read chapter 3, it looks like this is a one-way street where the
qualitative part validates regression assumptions).

This sounds convincing, but it should not come as a big surprise for
those who are familiar with [[nested
analysis](https://doi.org/10.1017/S0003055405051762){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}. The key distinction
underlying nested analysis is one between a causal effect and a causal
mechanism. The inference about a causal effect is different from an
inference about a mechanism, meaning that triangulation as Seawright
defines it is notpossible. It is true that the term 'triangulation' is
used here and there in the multimethod literature, for example by
Lieberman in his seminal APSR article (p. 440). However, it is not meant
in the way Seawright defines triangulation. The idea of [[triangulation
plays a bigger role in the mixed-methods
literature](http://dx.doi.org/10.1177/1558689812437187){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}, which is, as I said, largely
detached from the nested analysis literature. We do not know, but I
assume that researchers who might feel challenged by Seawright's main
claim are not among his readership, and that the political science
readers of *Multi-Method Social Science* need not be convinced that
integration is central to nested analysis.

Substantively, I think the issue is that the key selling point of
multimethod research has been made in Lieberman's pathbreaking article.
This is not to say that there is no room for methodological work on
nested analysis because some elements still have to be addressed and
refined. However, the one reason why integrated research is valuable,
the combined analysis of causal effects and mechanisms, can only be made
once and has been made already. I should add that the formulation of a
punchline is probably more difficult and less needed for methods books
than substantive books because the value of methods books lies in taking
one through the details of a method.

Stylistically, it is not obvious who is targeted with the rejection of
triangulated causal inference because there is no recent reference cited
in this context (1995 is the most reference that is given). This
reinforces my impression that the central claim is endorsed by everyone
because nobody in the nested analysis community argues for triangulation
and against integration. More generally, my reading of chapter 1 (and
the following chapters) is that it is a little bit too light on the
referencing side. In some cases, a reference jumps to mind that is not
cited, including Lieberman's article at several places, and in some
cases the literature is misrepresented (like the chain of references
supposedly saying that multimethod research is not superior to
single-method research, p. 2). I find this unfortunate because it limits
the value of the first chapter for empirical researchers who want to use
it as a guide to existing discussions of multimethod research and
further readings (the chapters do not have a "Further Readings" section
at the end).

Stay tuned for chapter 2 on the causal foundations of multimethod
analysis.


In *[Finding Pathways: Mixed-Method Research for Studying Causal
Mechanisms](http://www.cambridge.org/ve/academic/subjects/politics-international-relations/research-methods-politics/finding-pathways-mixed-method-research-studying-causal-mechanisms?format=PB){target="_blank"}*,
Weller and Barnes seek to explain "how the small-N component of
multi-method research can meaningfully contribute and add value to the
study of causal mechanisms" (quote from blurb). The book contains nine
chapters, including the introduction and concluding chapter. The second
chapter lays a foundation by discussing the basics and rationale of
multimethod research (MMR). Chapters 3 and 4 provide a very thorough
introduction of the case selection procedure that is proposed in the
book and a comparison of various case selection approaches. Chapter 5
continues with MMR on non-linear relationships, followed by chapter 6 on
using matching techniques for comparative case selection. Chapter 7
turns the tables by closing with a discussion of case selection that
elaborates on the benefits of running a regression analysis after the
case study. Chapter 8 summarizes the debate and takes a look forward to
future MMR studies.

 

*Finding Pathways* deals with a salient and widely discussed topic in
MMR. It is accessibly written and comes with all replication data online
(Stata do-files). This is a great benefit for users who seek guidance on
how to implement their own MMR. Substantively, the book raises several
interesting points that should guide the future debate about case
selection and process tracing in MMR. It extends the perspective to the
standard OLS model in the literature and considers non-linear
relationships and matching techniques (see also [Nielsen's recently
published
article](http://dx.doi.org/10.1177/0049124114547054){target="_blank"}).
Conceptually, it is also welcomed that they systemize potential settings
of mechanism that one might discern via process tracing which goes
beyond the usual consideration of a single causal chain.

 

These positive points of *Finding Pathways* are somewhat offset by
several problems as to the book's content and structure. First, there
are many redundancies pertaining to the presentation of Weller and
Barnes' preferred case selection strategy. It is described in detail in
chapter 2 and then again several times in the following chapters
combined with the presentation of specific case selection strategies.
The repetitious discussion of the general guidelines has little added
value and creates the impression that the key points could have been
delivered to the reader in a more succinct way.

 

Second, *Finding Pathways* does not establish a satisfying link between
the conceptual apparatus of the quantitative mediation literature and
the qualitative process tracing literature. In particular, chapters 8
and 9 indicate that Weller and Barnes' own background is quantitative.
This is not a problem per se and I believe there is potential for
linking research on mediation analysis with qualitative process tracing.
However, the link is not straightforward because many issues would have
to be sorted out first. Probably, the most important point concerns [the
definition of a
mechanism](https://ingorohlfing.wordpress.com/2012/03/20/mechanisms-in-qualitative-and-quantitative-research-2/){target="_blank"}.
In chapter 8, the discussion turns to the "average marginal effect of a
mechanism", i.e., an idea that would make many qualitative researchers
openly weep because mechanism and effects are understood to be
fundamentally different concepts.

 

Third, the book offers many interesting arguments, but it should have
linked to the existing literature more fully and developed some thoughts
further. With respect to the multi-method literature, it is striking
that [Lieberman's groundbreaking discussion of nested
analysis](http://dx.doi.org/10.1017/S0003055405051762) is not cited once
in the book. Weller and Barnes prefer the idea of pathway cases as
introduced by Seawright and Gerring and slightly modify it in one
respect. The pathway case does not figure in nested analysis,
highlighting that it would have been necessary to explain in detail why
the pathway case is superior to the strategy laid out in nested analysis
that is, much as the pathway case, in my understanding a type of what
Weller and Barnes call "residual-based case selection".

Another example of insufficient anchorage in the existing literature can
be found in the chapter on matching. The idea of using matching
techniques is very useful, but in the way it is introduced in *Finding
Pathways*, it is only a formalization of traditional case comparisons
discussed and used for decades in the qualitative domain. Seminal
contributions in this domain are completely ignored in this chapter.
Matching-based case selection therefore is not a genuine innovation, but
one step further in a decades-old development.

 

Despite these issues, *Finding Pathways* is a valuable addition to the
multimethod literature, offering many valuable insights; everyone who is
implementing and working on MMR should make use of this book.


The social science literature is full of discussions about causation and
what the best method for causal inference might be. However, a
relatively small percentage of them draw on the philosophical debate
about causation. Certainly, there is a great deal of talk about
philosophy of science on the ontological and metaphysical level,
including, for example, engagement with the relation between
neo-positivism and realism. What I have in mind are what are called
theories of causation in philosophy of science that ask what causation
is, what makes a causal statement true and how to detect a causal
relationship in practice.

The book, [[*Causation: A Very Short
Introduction*](http://ukcatalogue.oup.com/product/9780199684434.do){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}, by [[Stephen
Mumford](https://nottingham.ac.uk/philosophy/people/stephen.mumford){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} and [[Rani Lill
Anjum](https://sites.google.com/site/ranilillanjum/home){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}, is a useful entry point for
those who want to learn more about the theories that are on offer. The
book has ten chapters that are presented under the headings of "The
Problem", "Regularity", "Time and Space", "Necessity", "Counterfactual
Dependence", "Physicalism", "Pluralism", "Primitivism",
"Dispositionalism" and "Finding Causes". Chapters 2 through 9 are
conceptualizations of what causation is, while the last chapter
discusses strategies for finding causal relationships.

Those who are familiar with the debate in social science will notice
that the theories discussed in this book are more comprehensive, with
regularity and counterfactual dependence (supposedly) capturing
quantitative methods and physicalism (supposedly) covering process
tracing (e.g., Beach/Pedersen in their [[process tracing
book](http://www.press.umich.edu/2556282/process_tracing_methods){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}). Therefore, *Causation* is
certainly appropriate for extending one's horizon about theories of
causation.

Readers who expect in-depth introductions to a specific theory are not
well-served by the book, but this is not the goal of the [["Very Short
Introduction" series by Oxford
UP](http://ukcatalogue.oup.com/category/academic/series/general/vsi.do){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}. Beyond that, I had three
concerns when reading the book. First, the authors could have been more
generous with references to more specialized literature. At the end of
the book, the authors discuss a few books for each theory that they
discussed and offer their opinions on which is helpful, but for someone
who is accustomed to in-text citations referring to specific aspects,
this is not ideal.

Second, stylistically, the authors overstretch the use of questions
throughout the book. This is particularly problematic because I lost
sight of what questions were answered and which remained unaddressed.
Fewer questions and the use of summarizing tables would have been
beneficial in structuring the discussion.

Third, it was not always clear to me whether a theory was supposed to be
ontological or methodological, i.e., useful for the detection of causal
relationships. It became clearer when I reached the last chapter, which
is on causal inference (not a term that is widely used in philosophy of
science) and which mainly features [[Pearl's
account](http://www.cambridge.org/de/academic/subjects/philosophy/philosophy-science/causality?format=HB){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} and [[Woodward's
interventionist
theory](https://global.oup.com/academic/product/making-things-happen-9780195189537?cc=de&lang=en&){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}. (Although the interventionist
theory is close to the idea of an experiment, in my reading, Woodward
conceives of it as a semantic, conceptual theory, which makes it a bit
surprising that the authors present it as a methodological theory, a
problem that underscores the importance of clarifying what type of
theory one is talking about). Although the interventionist theory has
some appeal, I asked myself whether all metaphysical and ontological
theories are compatible with Woodward's and Pearl's account located on
the type level that has much affinity with quantitative, large-n
research. From a social science perspective, it would have been
interesting to read more about the link between metaphysical,
ontological and semantic theories on the one hand and methodological
theories on the other but that might have exceeded the goal for a *A
Very Short Introduction*.
