Ingo Rohlfing

At the end of last week, I came across a [blog entry by Olaf
Storbeck](http://economicsintelligence.com/2012/07/19/top-flight-german-business-prof-faces-severe-accusations-of-academic-misconduct/ "Storbeck: Lichtenthaler blog entry"){target="_blank"},
reporting that a rising star from German business economics, Prof.
Ulrich Lichtenthaler, is faced with numerous inquiries concerning his
publishing record. Two journals have already retracted three of his
articles and additional articles are under scrutiny. At present,
Lichtenthaler is confronted with three charges: unethical serial
publishing (or auto-plagiarism), inconsistent research across articles,
and sloppy statistical editing.

On the first count, journals criticize Lichtenthaler for publishing
articles that were substantively too close without cross-referencing his
own work. The journals argue that if they had known about his parallel
work, they would not have accepted his articles because of their lack of
originality. The second charge is that Lichtenthaler shows little
consistency in some empirical analyses. In substantively closely related
articles, he is using different variables in the empirical models
without good justification. This suggests that the models were tailored
so that they yield results amenable for publication.

The third point is a statistical one. Lichtenthaler's articles include
regression tables that assign a significant effect to variables that are
in fact non-significant. The point is not that data was made up. It is
simply that one finds a significance marker behind an estimated effect,
while the ratio of that effect to standard error shows that it does not
reach the corresponding level of significance. For example, in one
article (Lichtenthaler, U. and H. Ernst (2012): Integrated Knowledge
Exploitation: The Complementarity of Product Development and Technology
Licensing. Strategic Management Journal 33 (5): 513-534.), the variable
"technology licensing" has an effect of 0.69 and a standard error of
0.96 (OLS regression, model 14 on page 527). The t-value of about 0.72
is far from any conventional t-value used in empirical research, but the
effect is designated as significant at .10.

I think that each of the three points offers interesting implications
that hold beyond the field of business economics, as there is nothing in
this matter that is specific to this discipline. Starting with the third
point, the misreporting seems to be a matter of sloppiness because it is
easy to detect in principle (which does not make it any less serious an
issue). On the one hand, the misreporting could have been detected by
the reviewers. On the other hand, as a reviewer, one usually trusts in
the accuracy of the regression tables and focuses one's attention on
other parts of the paper. Now, the Lichtenthaler papers suggest that one
best look at the tables as well, but the significance of the effect is
not as easy to calculate as for individual variables in OLS regressions.
A labor-intensive alternative would be to have an editorial assistant
inspect all the tables of accepted publications for their accuracy.
However, the better alternative would be that journals require the
authors to publish their data and code alongside the article. It is hard
to understand why this is still the exception to the rule of not
publishing data as it would have many advantages beyond the opportunity
to check the accuracy of published work.

The other two points are very interesting because they are related to
somewhat more common practices in science. The second charge concerns
inconsistency of the theory and empirical models across publications. Of
course, one expects researchers to be consistent in their research and
to incorporate their previous findings into their current research.
However, I think Lichtenthaler's inconsistency is not that uncommon in
empirical research. At least, I personally have come across some
inconsistencies in social science publications and one wonders from time
to time why some publications fail to control for a variable that is
commonly included in empirical models. The Lichtenthaler case might
bring this issue to the forefront, at least in business economics, and
raise the bar in regards to the consistency and cumulative nature of
empirical research.

The first charge against Lichtenthaler, i.e., some variety of
auto-plagiarism, is related to the standard policy of journals to only
publish original work. I do not know the details, but it seems that
Lichtenthaler submitted very similar pieces to different journals at
similar points in time. If this was the case, the editors and reviewers
are not to blame because they could not know that multiple articles were
simultaneously under review. Lichtenthaler should have notified the
journals and, in case of doubt, send them all articles in order to let
them decide whether the respective submitted article still qualified as
original. Since this was not done, it only seemed a matter of time until
someone noticed the similarity between the articles. More generally,
auto-plagiarism is very likely to be detected ([as in the case of Bruno
Frey](http://economicsintelligence.com/2012/03/19/self-plagiarism-bruno-frey-gets-away-with-a-slap-on-the-wrist/ "Storbeck: Frey blog entry"){target="_blank"})
meaning that it is unlikely that one could succeed with this. However,
it would be better to prevent occurrences instead of retracting
articles. The only way I can think of (in addition to raising the
awareness for not doing this) would be a central registry of all
articles that are currently under review. This would allow editors to
determine whether the submitting researcher has another paper under
review that could be too close to the submitted work. If in doubt, the
editor could then approach the researcher and request clarification. In
addition to the question of whether all publishers would participate at
a central registry, it would shift the information balance between
editors and researchers more towards the editors because they
potentially get a full overview of all the work one currently has under
review. Publishers should ensure that their editors keep this
information absolutely confidential, but in my view, the shift of the
information balance is worth the price to avoid retractions of published
articles.

On a broader scale, this case indicates that Lichtenthaler either lacked
awareness of what is and is not legitimate in the publishing arena, or
he bent the rules in order to boost his output. In either case, this
incident underscores once more the direction in which that pressure to
publish has led science (and, in this particular instance, the field of
business economics). The pursuit of publications might have also led to
blindness of the field; Lichtenthaler has published about 50
peer-reviewed articles in about 8 years. This is not impossible, but it
should ring some warning bells (also because he argued that he did not
work more than 40 hours per week). This is not to blame the scientific
community for the Lichtenthaler case because he was the one who wrote
and submitted the articles. But the pressure for publishing creates an
environment that is conducive to such behavior and one should not be
surprised if similar incidents occur again.


At the end of last week, I came across a blog entry by Olaf Storbeck, reporting that a rising star from German business economics, Prof. Ulrich Lichtenthaler, is faced with numerous inquiries concerning his publishing record. Two journals have already retracted three of his articles and additional articles are under scrutiny.

Quality and quantity in publishing

The current version (vol. 10, issue 1) of the newsletter of the A[PSA
Section on Qualitative Methods and Multi-Methods
Research](http://www.maxwell.syr.edu/moynihan/cqrm/APSA_s_Qualitative_and_Multi-Method_Research_Section/ "APSA Section"){target="_blank"}
includes a symposium on John Gerring's 2^nd^ edition of [Social Science
Methodology](http://www.cambridge.org/us/knowledge/isbn/item6566290/?site_locale=en_US "Gerring: SSM2"){target="_blank"}
(SSM2, for short). The various contributors to the symposium deal with
different aspects of SSM2 such as concept formation (Ariel Ahram) or the
compatibility of set theory with the potential outcomes framework (by
Carsten Schneider and me, available on
[SSRN](http://ssrn.com/abstract=2096834 "Schneider/Rohlfing: Newsletter"){target="_blank"}).
Some of the contributors provided very strong criticism of Gerring's
book.  What I want to focus on here is Patrick T. Jackson's discussion
of Gerring's take on philosophy of science. To a degree, I have to admit
that it was a fun read for me. Jackson starts out by saying that he
wholeheartedly agrees with the last paragraph of the book, but disagrees
with most of what is written on all the previous pages. Why is that?

At the risk of oversimplification, Gerring argues that social science
researchers should be able to communicate with each other. Communication
is hampered by the pursuit of different ontologies and understandings of
science (Gerring frequently speaks of methodology, but I think he mostly
means ontology and sometimes epistemology). According to Gerring, this
implies that the adoption of one methodology and an understanding of
science is necessary for effective communication, and this should be the
(neo-)positivist take on science. Science then should be "systematic,
rigorous, evidence-based, falsifiable, replicable, generalizable,
non-subjective, transparent, skeptical, rational, frequently causal, and
cumulative" (p. 2). Jackson correctly asserts that, for one reason or
another, many social scientists do not subscribe to this view and are
likely to be offended at calling these the characteristics of true
science. He criticizes Gerring's plea for unity from various sides: the
definition of causal mechanisms; the reference to falsifiability, an
idea that one usually shies away; and the need for unity in the first
place, just to mention three points.

However, there is a more fundamental problem with Gerring's plea for
unity that is not addressed by Jackson. As Gerring notes (and what is
not criticized by Jackson), ontological disputes cannot be settled
because they are beyond the reach of empirical analysis (see alsoHay,
Colin (2006): Political Ontology. Goodin, Robert E. and Charles Tilly
(eds.): *The Oxford Handbook of Contextual Political Analysis*. Oxford:
Oxford University Press: 78-96). This is what Gerring identifies as a
problem because exchange among scholars is hampered by different
ontological and, as a consequence, methodological positions. That's all
fine, but Gerring then draws an unsustainable conclusion from this: that
all researchers should follow one ontology and that this should be his
pet ontology of the neo-positivist sort. I think the problem with this
claim is apparent: if there is no right and wrong ontology and we cannot
adjudicate between them, how do we justify all researchers settling on
one of these ontologies? Regardless of whether one likes Gerring's view
of science and ontology or not (Jackson clearly is on the do-not-like
side), one should eschew claiming that one has "found" an ontology that
everyone should follow. For the same reason, there is little to be
gained by criticizing Gerring's view on science (or anyone's view on
science) by censuring it on substantive ground. One might disagree with
Gerring's conception of science and ontology, but these criticisms are
not any more right or wrong than Gerring's position itself.

Having said that, we are still left with Gerring's correct diagnosis
that the pursuit of different ontologies hampers communication (also
partly because many scholars treat ontology as a matter that is
decidable, therefore trying to convince the "disbelievers" of the
"truth" that they own). What then is the basis for judging empirical
insights gathered on the basis of a specific ontology? In my view, the
answer is *consistency*, which has two elements. First, one's personal
take on ontology, methodology, and epistemology should form a coherent
whole (see Petter Hall's (2003: Aligning Ontology and Methodology in
Comparative Research. Mahoney, James and Dietrich Rueschemeyer (eds.):
*Comparative Historical Analysis in the Social Sciences*. Cambridge:
Cambridge University Press: 373-404) intriguing piece on aligning
ontology and methodology). Second, each of the three "-ologies" should
be internally consistent. The rationale for consistency as the guiding
principle is that if we cannot judge the adequacy of an ontology (and
the other --ologies), we should focus on the extent to which a
researcher does justice to his/her ontology and follows a methodology
and epistemology that is consistent with it.

Here is a simple example of what consistency means. When you run a
regression, you automatically make the assumption that the cases are
causally homogenous (the assumption can be right or wrong, but it is the
default assumption with which one starts). Now, suppose that you do
post-regression process tracing in order to learn something about the
underlying causal mechanisms. One sometimes finds the opinion that the
generalization of insights derived from process tracing should follow
the idea of contingent generalization (Nome, Martin Austvoll (2008):
Getting the Pathway Case Off the Drawing Board. *APSA Qualitative
Methods Newsletter* 6 (1): 20-23). Contingent generalization means that
causal inferences are only generalized to cases that share certain
conditions. For instance, when the regression analysis includes all
countries in the world, contingent generalization would mean that
process tracing insights gathered in a European country are only
generalized to other countries from Europe. This practice is
inconsistent because it is in discord with the causal homogeneity
assumption embodied in the regression analysis. One either has to limit
the regression analysis to European countries, or go for a more sweeping
generalization of the process tracing inferences. One can think of more
severe consequences of inconsistency than this one, but it illustrates
the importance of thinking coherently when linking ontology,
methodology, and epistemology to each other.

I acknowledge that it can be painful to take ontologies for granted,
ontologies with which one wholeheartedly disagrees, and to limit oneself
to an assessment of their internal coherence and consistency with
methodology and epistemology. But ontology wars based on the flat
rejection of any ontology other than the one we prefer yield little
benefit. At the least, the engagement with consistency hopefully ensures
that researchers produce the best research *given* their ontological
commitments.


Live and let live, or: unity and diversity in the social sciences

Alt-Bundespräsident Roman Herzog wird [von Focus mit den Worten
zitiert](https://www.focus.de/politik/deutschland/der-altbundespraesident-im-focus-interview-roman-herzog-will-fuenf-prozent-huerde-reformieren_aid_751665.html "Herzog Focus-Interview"){target="_blank"}:
„Im Prinzip ist die Fünf-Prozent-Hürde nicht mehr zeitgemäß. Eigentlich
müssten wir die Hürde nach oben setzen." Die Begründung: Vor dem
Hintergrund von immer mehr kleineren Parteien werde der
Bundeskanzler/die Bundeskanzlerin ansonsten „nicht mehr von einer großen
Mehrheit der Bevölkerung getragen". Dies sieht Herzog als eine
Gefährdung der parlamentarischen Demokratie.

Das gesamte Interview kenne ich nicht und die Aussagen mögen aus dem
Kontext genommen sein. Vorbehaltlich dessen: Die Aussagen von Roman
Herzog haben aus mehreren Gründen Schieflage. Erstens, der Bundeskanzler
bzw. die Bundeskanzlerin sind noch nie von einer Mehrheit der
Bevölkerung getragen worden, wenn „getragen" gleichbedeutend ist mit
„gewählt". Aufgrund von Nichtwählern hatten Koalitionen auf Bundesebene
bis dato die Mehrheit der Wähler hinter sich, aber nicht, zumindest
nicht explizit, die Mehrheit der Bevölkerung. (Selbst in Bayern hatte
die CSU nie die absolute Mehrheit der Wähler hinter sich.)

Zweitens, wenn mehr als zwei Parteien eine Mehrheitskoalition eingehen
steht zumindest noch die Hälfte der Wähler hinter der Regierung.
Sicherlich sind Dreiparteienkoalitionen in der Tendenz konfliktreicher
und potenziell instabiler als Ein- und Zweiparteienregierungen. Aber die
Diskussionen über die Ampel und Jamaika-Koalitionen zeigt, dass die
Parteien dies bereits in Erwägung ziehen (im Saarland ist Jamaika nicht
an Koalitionsfragen gescheitert, sondern an internen Problemen der
FDP).  Es ist somit nicht offensichtlich, wieso die parlamentarische
Demokratie gefährdet sei.

Drittens, die Prozenthürde ist ein zweischneidiges Schwert und eine
höhere Hürde hat ihre Tücken. Die Hürde soll einer Zersplitterung des
Parteiensystems entgegenwirken und eine gewisse Stabilität
sicherstellen. Gleichzeitig führt die Hürde zu einer Disproportionalität
von Stimmanteil und Sitzanteil. Je mehr Parteien die Hürde reißen, umso
mehr profitieren die Parteien davon, die die Hürde überspringen, da ihr
Sitzanteil im Parlament den Stimmanteil zunehmend übersteigt. Umso höher
die Hürde ist, umso mehr Stimmen gehen verloren und umso mehr Wähler
sind nicht durch Parteien im Parlament repräsentiert. Wenn man
Stabilität, verstanden als eine geringe Anzahl von Parlamentsparteien,
wichtig findet mag man dies in Kauf nehmen. Die parlamentarische
Demokratie lebt aber wesentlich von Repräsentation und die wird durch
eine hohe Hürde untergraben. Eine hohe Wahlhürde ist daher mindestens
ebenso gefährlich für die parlamentarische Demokratie wie, wenn man
Herzog folgen mag, eine niedrige.

Viertens, die steigende Anzahl an Parteien ist Ausdruck
gesellschaftlicher Vielfalt und Entwicklungen. Wenn sich neue Interessen
in einer Gesellschaft ausbilden und die etablierten Parteien diese nicht
abbilden, nehmen Parteineugründungen eine wichtige Funktion wahr, weil
sie diese Interessen repräsentieren und in das Parlament einbringen. Wer
neuen Interessen die parlamentarische Repräsentation verwehrt, indem man
die Wahlhürde hoch ansetzt, ignoriert gesellschaftliche Entwicklungen.
Bestenfalls geben hohe Hürden den etablierten Parteien mehr Zeit, auf
neue gesellschaftliche Trends zu reagieren und diese in die eigene
Programmatik aufzunehmen. Im schlechtesten Fall schlägt die
Privilegierung der etablierten Parteien in Frust um, die sich in
niedriger Wahlbeteiligung oder Zulauf für radikale Parteien
niederschlägt.


Alt-Bundespräsident Roman Herzog wird von Focus mit den Worten zitiert: „Im Prinzip ist die Fünf-Prozent-Hürde nicht mehr zeitgemäß. Eigentlich müssten wir die Hürde nach oben setzen.“ Die Begründung: Vor dem Hintergrund von immer mehr kleineren Parteien werde der Bundeskanzler/die Bundeskanzlerin ansonsten „nicht mehr von einer großen Mehrheit der Bevölkerung getragen“. Dies sieht Herzog als eine Gefährdung der parlamentarischen Demokratie.

Roman Herzog und die Hürde

Recently, [Frontal
21](http://frontal21.zdf.de/ZDFde/inhalt/1/0,1872,1001633,00.html?dr=1 "Frontal 21"){target="_blank"}
-- a German newscast -- reported about the handling of error in
treatments under German health care law (about which I am not an
expert). A lawyer specializing in health care law explained that it is
very difficult to win lawsuits against medical doctors (MD's) because
the patient claiming to experience treatment error has to substantiate
this claim (except for grave treatment error). Particularly problematic
are cases where patients die due to treatment error and the patient
suffered from a disease that would cause death if left untreated. MD's
charged of treatment error argue that the patient would have died anyway
and that the alleged error is not causal for the patient's death.
According to the report, this interpretation is supported by German
health care law, meaning that lawsuits have little chance of successful
litigation.

This handling of treatment errors touches on two prominent issues from
the philosophy of science: *overdetermined events* and *contrasts* or
*counterfactuals*. On the first point, the question is: what is the
cause when two events can bring about the outcome at the same or
different points in time? An example from the philosophy of science
literature takes the assassination of a person as the outcome (e.g.,
Sartorio, Carolina (2005): Causes as Difference-Makers. Philosophical
Studies 123: 71-96). Imagine two assassins operate independently. One
assassin opts for poisoning, while the other opts for a sniper assault.
The first assassin poisons the target, but before the target dies, the
second sniper shoots the individual (sometimes, you have stories like
this in movies or TV series like CSI). This is an instance of
overdetermining causes because the poison and the shooting alone would
have sufficed to kill the target.

Now, is the second assassin and the shooting the cause of the victim's
death? If one is prepared to infer causation from a chain of events -- a
mechanism, if you like -- the answer would be yes because the bullet
fired by the sniper terminates the target's life (e.g., Schaffer,
Jonathan (2003): Overdetermining Causes. Philosophical Studies 114
(1-2): 23-46). But it is less straightforward from a counterfactual
perspective because if the shooter had missed the target, the person
would nevertheless have died from the poison. In this naïve view, the
answer would be no. The question of what a cause is when one is
confronted with overdetermined causes is not settled by the philosophy
of science literature. Apparently, however, German health care law takes
a simple counterfactual view on causation because it denies error in
treatment as a cause of the patient's death if the patient suffered from
a deadly disease.

The second interesting aspect is the question of what the relevant
contrast or counterfactual is. Health care law compares the factual
world in which the patient died because of an error in treatment with a
counterfactual world in which the patient does not receive any treatment
and dies from the deadly disease. Personally, I was a little bit struck
by this because the patient receives treatment in order to be cured from
the disease. An alternative, at least equally plausible counterfactual
for the world in which error leads to death is one where no error is
committed and the patient is cured. This counterfactual would make it
easier to charge MD's with error, while the counterfactual that is made
in health care law favors the MD. I am not making any substantive plea
for either counterfactual here, but simply note the relevancy of topics
discussed in the philosophy of science for everyday life (or death, in
this case).


Recently, Frontal 21 – a German newscast – reported about the handling of error in treatments under German health care law (about which I am not an expert). A lawyer specializing in health care law explained that it is very difficult to win lawsuits against medical doctors (MD’s) because the patient claiming to experience treatment error has to substantiate this claim (except for grave treatment error). Particularly problematic are cases where

Error in medical treatment and causation in German health care law

The current APSA Comparative Politics Newsletter is dedicated to "Doing
Comparative Politics Elsewhere" (i.e. , outside of the US). [Thomas
Plümper](http://www.polsci.org/pluemper/ "Thomas Plümper"){target="_blank"}
contributes a discussion on Comparative Politics in Europe. In brief,
Plümper argues that, until recently, the field of Comparative Politics
(CP) has been dominated by qualitative methods. During the last ten
years or so, Plümper has seen the advent of sophisticated quantitative
research, as it has been practiced for decades in the United States.

The prevalence of small-n research is attributed to Lijphart's 1971
article published in APSR (Lijphart, Arend (1971): Comparative Politics
and the Comparative Method. *American Political Science Review* 65 (3):
682-693). The article was widely perceived (about 360 citations
according to the SSCI as of today) and distinguished between the
experimental, statistical, comparative, and case study method in the
social sciences. The comparative method is effectively used synonymously
with qualitative comparative case studies, the case study method being
confined to single-case studies. The mainstream method in the US became
the statistical method, while Europe followed the comparative trail
(experiments only recently becoming more popular on both sides of the
Atlantic).

Plümper does not find Lijphart's distinction convincing and does not
understand why the comparative method became so dominant. He argues that
the number of cases is not exogenously set, but depends on the theory
and hypotheses at hand. If theory is the guide, one should sometimes do
qualitative research, but should also occasionally do quantitative
research. The (almost) exclusive realization of case studies in Europe
indicates that theory did not always drive the design and method, but
that theory was fit to the need to do qualitative research. Hard figures
on whether US research was largely quantitative since the 1970s and
European research qualitative are unavailable, but it seems to me that
Plümper's perception is shared by many.

Accepting this (now narrowing) divide across the Atlantic, I would add
three observations concerning Plümper's diagnosis. The first is related
to Lijphart's article and his four-fold distinction; the second to the
methods in the US; and the third to the theory-design link in empirical
research.

[Lijphart and the comparative
method]{style="text-decoration:underline;"}

There are many different ways to describe the field of social science
methods. My hunch is that, at present, most people see the major line of
division between experimental and non-experimental, i.e., observational
research (this presumes that we, as Lijphart did, only discuss
"neo-positivism," without any intention of degrading interpretivist
research). Within the observational camp, one major line of division is
between quantitative/large-n and qualitative/small-n research. Indeed,
then, it is peculiar to put experimental, statistical, comparative
methods in one row because the former two are also comparative and it is
hard not to be comparative at all (Lees, Charles (2006): We Are All
Comparativists Now: Why and How Single-Country Scholarship Must Adapt
and Incorporate the Comparative Politics Approach. *Comparative
Political Studies* 39 (9): 1084-1108). In retrospect, it is also odd how
much debate there was about the comparative method in the 1970s and
1980s (e.g., DeFelice, E. Gene (1980): Comparison Misconceived -- Common
Nonsense in Comparative Politics. *Comparative Politics* 13 (1):
119-126; Frendreis, John (1983): Explanation of Variation and Detection
of Covariation: The Purpose and Logic of Comparative Analysis.
*Comparative Political Studies* 16: 255-272).

Nevertheless, Lijphart's discussion of the comparative method (and case
study method) is seminal because it clearly identifies major problems
for causal inference as well as some remedies. However, this should not
obscure that Lijphart actually declares the comparative method to be
weaker than experimental and statistical methods (p. 685). If it were
true that European scholars were influenced by Lijphart's distinction of
methods, they would have missed the point because they opted for a weak
method.

[The statistical method and the US]{style="text-decoration:underline;"}

Second, the development of CP in the US was more in line with Lijphart
because the mainstream method was (and is) the quantitative one. If CP
in Europe was one-sidedly qualitative, though, CP in the US was
one-sidedly quantitative, thus also indicating theory's lack of
influence on the design and choice of method.

Interestingly, we see converging trends on both sides of the Atlantic.
In the US, the dominance of quantitative methods gave rise to the
foundation of the [APSA
Section](http://www.maxwell.syr.edu/moynihan/cqrm/APSA_s_Qualitative_and_Multi-Method_Research_Section/ "APSA Section"){target="_blank"}
on Qualitative Methods, recently renamed the Section on Qualitative
Methods and Multi-Method Research. In Europe, we now have the [European
Political Science
Association](http://www.epsanet.org/ "EPSA"){target="_blank"} (EPSA),
seeking the promotion of more sophisticated formal and quantitative
research on this side of the Atlantic. These are laudable developments
that reflect Plümper's claim that qualitative and quantitative methods
both have their place in CP.

[The dependence of theory on
methods]{style="text-decoration:underline;"}

Third and related to the previous point, it is correct that the number
of cases is not exogenous. The state of the field and theory should
determine whether to do a quantitative or a qualitative analysis
involving many or a small number of cases. From a sociology of science
perspective, however, this only shifts the problem to the theoretical
part of the analysis because the theoretical goal is not necessarily
exogenous to a researcher's take on the small-n/large-n question. If a
researcher is (implicitly or explicitly) predisposed to case studies,
she formulates a theoretical goal and hypotheses that are better
examined with case studies. The reverse holds for a researcher inclined
to do quantitative analyses. In this perspective, theory does not drive
the choice of the method, but rather methodological predispositions
determine the theoretical goal of a study.

In the best of all (social science) worlds, this is not a huge problem
as long as researchers that only do quantitative or qualitative analyses
accept that the respective other method has a role to play as well and
acknowledge the results produced with that method. In the worst of all
worlds, researchers attached to one method negate the value of the other
method. The increasing advocacy for multi-method research is a promising
sign that we are closer to the best than the worst of all worlds that
one can imagine for the social sciences.


The current APSA Comparative Politics Newsletter is dedicated to “Doing Comparative Politics Elsewhere” (i.e. , outside of the US). Thomas Plümper contributes a discussion on Comparative Politics in Europe. In brief, Plümper argues that, until recently, the field of Comparative Politics (CP) has been dominated by qualitative methods.

Comparative politics and the choice of methods

Research on causal mechanisms is a growth industry, with the largest
percentage of studies falling into the camp of qualitative research. The
reason for this is the admonition that correlation is not causation,
implying the claim that valid causal inference requires evidence on
causal mechanisms. While the need for causal mechanisms is occasionally
contested, there is an increasing consensus that mechanisms have their
place in empirical research and causal inference.

Although there is no commonly held understanding of what a mechanism is,
there is at least broad agreement that mechanisms ensure productive
continuity between a macro cause and a macro outcome, which implies that
mechanisms are located on a lower level of analysis. Thus, the analysis
of mechanisms is considered to be the key feature of qualitative case
studies and process tracing. Quantitative research, in contrast, is
focused on the estimation of causal effects on the macro level.

To a degree, a recent publication by Imai, Keele, Tingley, and Yamamoto
(Imai, Kosuke, Luke Keele, Dustin Tingley and Teppei Yamamoto (2011):
Unpacking the Black Box of Causality: Learning About Causal Mechanisms
from Experimental and Observational Studies. *American Political Science
Review* 105 (4): 765-789) addresses this division of labor between
qualitative and quantitative research. Imai et al. develop further
mediation analysis in order to allow for the search for causal
mechanisms in quantitative research. The following figure illustrates
the most basic setting of mediation analysis. A treatment (cause) T
works through a mechanism (M), equivalent to the mediator, which is, in
turn, tied to the outcome (O). Moreover, the treatment has a direct
effect on the outcome. Mediation analysis allows one to estimate both
the direct and indirect effects, and the effect of the treatment on the
mechanism and of the mechanism on the outcome.

[![](https://ingorohlfing.files.wordpress.com/2012/03/mechanisms-chart1.jpg?w=300&h=225 "Mechanisms - Chart"){.aligncenter
.size-medium .wp-image-367 loading="lazy" attachment-id="367"
permalink="https://ingorohlfing.wordpress.com/2012/03/20/mechanisms-in-qualitative-and-quantitative-research-2/mechanisms-chart-4/"
orig-file="https://ingorohlfing.files.wordpress.com/2012/03/mechanisms-chart1.jpg"
orig-size="960,720" comments-opened="1"
image-meta="{\"aperture\":\"0\",\"credit\":\"\",\"camera\":\"\",\"caption\":\"\",\"created_timestamp\":\"0\",\"copyright\":\"\",\"focal_length\":\"0\",\"iso\":\"0\",\"shutter_speed\":\"0\",\"title\":\"\"}"
image-title="Mechanisms – Chart" image-description="" image-caption=""
medium-file="https://ingorohlfing.files.wordpress.com/2012/03/mechanisms-chart1.jpg?w=300"
large-file="https://ingorohlfing.files.wordpress.com/2012/03/mechanisms-chart1.jpg?w=960"
width="300" height="225"
srcset="https://ingorohlfing.files.wordpress.com/2012/03/mechanisms-chart1.jpg?w=300 300w, https://ingorohlfing.files.wordpress.com/2012/03/mechanisms-chart1.jpg?w=600 600w, https://ingorohlfing.files.wordpress.com/2012/03/mechanisms-chart1.jpg?w=150 150w"
sizes="(max-width: 300px) 100vw, 300px"}](http://ingorohlfing.files.wordpress.com/2012/03/mechanisms-chart1.jpg)

At first glance, a qualitative researcher might be skeptical that
mediation analysis is really about mechanisms. I suppose that one
counterargument would be that mediation analysis is still about the
estimation of effects and relies on what we currently call data set
observations. This contrasts with process tracing, which is closely tied
to the idea of causal process observations and the gathering of a rich
body of diverse pieces of evidence. Thus, the warning that correlation
is not causation would apply in mediation analysis as well, whereas it
would now pertain to the link between the treatment and the mediator,
and the mediator and the outcome. Moreover, one might believe that the
distinction between an indirect and a direct effect does not resemble
anything we know from case studies, prompting the conclusion that the
quantitative understanding of mechanisms does not overlap its
qualitative counterpart.

However, this would all too easily dismiss mediation analysis. Two
issues deserve closer scrutiny. First, reference to a 'direct effect'
does not imply that the effect of the treatment is indeed direct, that
is, it functions without a mechanism. The blackboxing of the direct
effect simply denotes that one is only interested in the mechanisms that
one models via the mediator. Thus, the direct effect also works via
mechanisms that are not, however, theoretically relevant. Second, Imai
et al.'s definition of mechanisms is similar to some of the definitions
found in the qualitative literature. Moreover, the two empirical
examples that they present include a mechanism that is located on a
lower level than the treatment and the outcome, therefore meeting the
common understanding in this respect. Third, one might reject the notion
of a mechanism having an effect because the concept of a mechanism often
serves as an antipode to the idea of an effect. However, in philosophy
of science, one also finds some taking the position that mechanisms have
effects (Pearl, Judea (2009): *Causality: Models, Reasoning, and
Inference*. 2^nd^ ed. Cambridge: Cambridge University Press.), meaning
that Imai et al. are not exploring entirely new ground here. The reason
that we might resist the idea of a mechanism having a certain effect is
that we usually do not have the data to scrutinize the mechanism via a
large-n analysis, requiring it to perform process tracing. Seen from the
other side, however, this would mean that with sufficient data, one
could estimate the effect attached to a mechanism (as Imai et al. do).
Seen in this light, the link between the concepts of mechanisms and
effects might be closer than is apparent at first glance.

Even if one were to accept these arguments, there remains the criticism
that a significant effect of a mechanism does not resemble causation
because the treatment might not be connected to the mediator, and/or the
mediator may not relate to the outcome. Thus, even if one accepts that
mechanisms can be analyzed quantitatively, there is, of course, still
room for process tracing as one needs to discern productive continuity
between treatment, mediator, and outcome (which might give rise to
problems of infinite regress). But I presume that qualitative
researchers see more problems with the quantitative take on mechanisms,
among other reasons because of the neat distinction between effects on
the one hand and mechanisms on the other. As Imai's work is rather new,
I expect a great many interesting discussions to be ahead.


Research on causal mechanisms is a growth industry, with the largest percentage of studies falling into the camp of qualitative research. The reason for this is the admonition that correlation is not causation, implying the claim that valid causal inference requires evidence on causal mechanisms.

Rogue Scholar Beiträge

Quality and quantity in publishing

Live and let live, or: unity and diversity in the social sciences

Roman Herzog und die Hürde

Error in medical treatment and causation in German health care law

Comparative politics and the choice of methods

Mechanisms in qualitative and quantitative research