Ingo Rohlfing

About two weeks ago, COMPASSS issued a [Statement on Rejecting Article
Submissions because of QCA Solution
Type](http://www.compasss.org/files/compasss-rejection-policy-statement-20170814.html){target="_blank"
rel="noopener"}. In short, the reasoning was that methodological work on
QCA is developing and that reviewers and editors should not judge
empirical work based on whether one particular solution type is
interpreted as causal. (*Disclosure: I am a member of the COMPASSS
advisory board, but was not involved in this statement in any way*.)
Dimiter Toshkov picked up on the statement and wrote an interesting
[blog post](http://re-design.dimiter.eu/?p=905){target="_blank"
rel="noopener"} on this. Dimiter's blog then hosted a [response by Eva
Thomann](http://re-design.dimiter.eu/?p=915&){target="_blank"
rel="noopener"} that led to a say, heated, exchange of arguments in the
comments section.

With some delay, I want to raise two points about the debate that
largely abstracts from the question of which type of QCA solution
(conservative, intermediate, parsimonious) is the "correct" one and can
be interpreted as causal. I have my viewpoint on this, but my remarks
are more general.

**The bar for public statements should be high**

It is not without precedent that professional associations, which
COMPASSS is for me, publish statements on various matters (such as the
[ASA statement on
p-values](https://www.google.de/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwjdn_-H8cfWAhUHZVAKHYNeDd0QFggyMAA&url=http%3A%2F%2Fwww.amstat.org%2Fasa%2Ffiles%2Fpdfs%2FP-ValueStatement.pdf&usg=AFQjCNF8mOLti7RmBKyHaks6dpjqcot8Nw){target="_blank"
rel="noopener"}). I believe that associations have the right to issue
statements and should do so. However, the bar should be set high
because, otherwise, there may be interference with the scientific
process, with the association tending to take sides in a debate (not
necessarily, though). Most of the time, the association should be a
bystander and not an involved party.

The question of how to review QCA submissions to journals certainly is
an important topic that potentially qualifies for a statement. In light
of Eva Thomann's post, however, my impression is that the bar has been
set too low. She writes that the COMPASSS Steering Committee (SC) had no
information about how often papers have been rejected based on arguments
that a specific solution type can(not) be interpreted as causal. One
might read this such that the issue is so important to the SC that the
frequency of its occurrence does not matter. However, I am far from
convinced that frequency should be irrelevant.

If we take this as the precedent of COMPASSS statements, it would mean
that the SC could put out statements at a high rate because I am sure
that reviews of QCA articles often contain arguments the members of the
SC consider dubious or wrong, some of which strike me as being worse
than the issue at stake here. As many QCA researchers surely can tell,
one still has to manage reviews that flatly deny any value of QCA. One
might respond that the SC saw a problem emerging and wanted to intervene
early on, but I do not find this convincing. As a professional
association, COMPASSS includes respected QCA researchers and experts,
but it would still mean that the only professional association in the
field would intervene and, further, would need to frequently intervene
in the future because reviewer arguments that it deems ill-suited will
continue to be made.

**Most studies interpret one solution type as causal**

The second point concerns the internal coherence of the argument, which
dips into the question of which of the three solution types is causal.
The COMPASSS statement does not say anything about a specific type, but
readers with some knowledge of the field of QCA can reasonably guess
that it is about Baumgartner's and Thiem's work and argument that only
the parsimonious solution should be interpreted as
causal\[[1](http://doi.org/10.1007/s11135-014-0026-7){target="_blank"
rel="noopener"},
[2](https://doi.org/10.1177/0049124117701487){target="_blank"
rel="noopener"}\]. If you follow their reasoning, the question is: why
should reviewers not be allowed to criticize a study if it interprets
another type as causal? I agree that the Baumgartner/Thiem article is
"only" one study and that more work is needed on this. However, this is
not a sufficient reason to discount the work that has been done and it
should mean something. If this was the standard, we could pretty much
stop doing research because one could always say "I don't follow this
reasoning because there is not enough research about it." (We would have
to define what "enough" means, but this is probably even more subjective
than the question of QCA solution types.)

Now to the internal consistency of the argument: There seems to be
consensus that only one of the three solution types can be interpreted
as causal (this should be consensus). This means that as soon as an
empirical researcher interprets any solution type as causal (as many
do), they are implicitly saying that the other two types are not causal.
You might, like Eva, say other types are insightful or useful, but they
cannot be causal (another point on which Eva seems to agree). The
Baumgartner/Thiem argument that only one solution type is causal is
therefore *not new at all* (the content is, I'd say, because most
empirical researchers now seem to prefer the intermediate solution).

This means you can criticize reviewers for rejecting papers because they
interpret the wrong type as causal, "wrong" from the reviewer's
perspective. For the sake of consistency, you should then also criticize
empirical researchers if they interpret only one solution type as causal
because they are designating the other solution types as non-causal. The
liberal attitude that shines through the COMPASSS statement, stating
that any solution could be interpreted as causal, is not sustainable
because the causal interpretation of any solution type is exclusive.
Either reviewers should be allowed to settle on the causal
interpretation of one solution type, or reviewers *and authors* should
not interpret any type as causal.

**A way out**

Avoiding causal terminology might not be the worst of ideas, but I think
there is a middle way between rejecting papers because of the wrong
solution type and not using any causal terminology per se. I agree with
Dimiter and Eva that rejecting a paper based solely on the question of
solution types should be discouraged. This has nothing to do with the
topic, but rather the understanding of what constitutes good reviewer
practice. If you think the paper is good, but also that the wrong
solution type is interpreted as causal, then ask the author to clarify
their position on why this solution type is taken as causal. Perhaps the
author has a good answer for why this type and not another and the issue
can be settled. Or the author starts thinking, accepts that this
solution might not be interpreted as causal, and switches to non-causal
terminology. I believe that most QCA researchers want to make causal
claims, but we should keep in mind that good and interesting research
can also be non-causal.


About two weeks ago, COMPASSS issued a Statement on Rejecting Article Submissions because of QCA Solution Type. In short, the reasoning was that methodological work on QCA is developing and that reviewers and editors should not judge empirical work based on whether one particular solution type is interpreted as causal.

The COMPASSS statement and QCA solution types

How to properly interpret consistency and coverage values in fuzzy-set QCA.

In 2000, Ragin added the idea of fuzzy sets to Qualitative Comparative
Analysis (QCA) and they have been widely used since then. However, one
sometimes still finds misunderstandings about what fuzzy sets are, in
particular when it comes to the interpretation of consistency and
coverage scores.

In a criticism of QCA by Stockemer (which is criticized for a variety of
reasons and to which Stockemer again replied), he runs a fuzzy-set
analysis. The outcome is 'high levels of representation of women in a
parliament' which is not important here. The solution coverage is 0.23,
which leads Stockemer to conclude that "the combined solution only
covers 23 per cent of all cases" (p. 94).

This statement is not true for fuzzy-set QCA (and would neither be true
for crisp-set QCA). For fuzzy sets, solution coverage measures the
percentage of the sum of the outcome membership values that are covered
by the consistent cases' membership in the solution. As most cases have
a membership of less than 1 in the outcome and the solution, this
usually is unrelated to any case-related percentages. If we look at the
data, we see that the analysis includes 54 cases in total. Seven of
these 54 cases are members of the solution. Five cases are typical, one
is a deviant case consistency in degree and one is deviant in kind. If
these cases are meant by "combined solution only covers 23 per of all
cases", the highest percentage we get is 13 (7/54). If we limit the
perspective to the members of the outcome, that is, we speak of coverage
for crisp-set QCA now, we have a coverage value of 0.3 (6/20). (I leave
aside here that one should not simply switch to a crisp-set perspective
based on fuzzy-set results.) If we additionally limit it to the typical
cases, we have a coverage value of 0.26 (5/19, because one member of the
solution is not a typical case). Regardless of how coverage is
calculated if not in the proper way, we do not get a coverage value of
0.23.

It is relatively easy to be on the safe side when interpreting
consistency and coverage scores. Just interpret them without any metric,
for example, by saying "Solution consistency is 0.34" or "Raw coverage
of the first term in the solution is 0.23". This might be less intuitive
than speaking of percentages of cases or percentages more generally, but
it is the proper reading of the scores and avoids misunderstandings.


In 2000, Ragin added the idea of fuzzy sets to Qualitative Comparative Analysis (QCA) and they have been widely used since then. However, one sometimes still finds misunderstandings about what fuzzy sets are, in particular when it comes to the interpretation of consistency and coverage scores. In a criticism of QCA by Stockemer (which is criticized for a variety of reasons and to which Stockemer again replied), he runs a fuzzy-set analysis.

Don’t conflate fuzzy set membership with cases in QCA

Qualitative Comparative Analysis (QCA) is the method of choice for the
analysis of set relations and has changed considerably and improved over
the years. The more one delves into the method, however, the more things
you (I, at least) stumble upon that seem a little bit curious, sometimes
also more than a bit (just as with any method, I guess). One issue I
came across relates to the calculation of consistency scores in
fuzzy-set QCA for assigning outcome values to truth table rows.

I used data from [[a study about parliamentary control of EU
affairs](http://dx.doi.org/10.1080/01402382.2012.706412){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}, but the specific study and
data are not important because it can happen with any data. I noticed
that there is a truth table row, row 6 by the automatic counting of the
[[QCAGUI package for
R](https://cran.r-project.org/web/packages/QCAGUI/index.html){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}, which only has the Czech
Republic as a member. The consistency value of the row is 0.78 and could
be included in the minimization process because it is above the
conventional minimum level of 0.75. The author of the study imposes a
higher threshold than 0.78, but this is not relevant because it only
matters that it legitimately could have been taken as consistent with
the claim that the configuration is sufficient.

Now, when you take a look at the XY-plot for this truth table row, you
see that the Czech Republic is a contradictory case in a qualitative
view (or a "true logical contradiction", as
[[Schneider/Wagemann](http://www.cambridge.org/de/academic/subjects/politics-international-relations/research-methods-politics/set-theoretic-methods-social-sciences-guide-qualitative-comparative-analysis?format=PB){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} call it). This means the
consistency level would allow you to include the row in the minimization
process, but the only member of the configuration is not a member of the
outcome. In other words, the row would be taken as sufficient, but you
could not point to or study a single case representing the sufficient
relationship.

![row6.png](https://ingorohlfing.files.wordpress.com/2016/06/row6.png){.size-full
.wp-image-970 .aligncenter loading="lazy" attachment-id="970"
permalink="https://ingorohlfing.wordpress.com/2016/06/21/curiosities-of-qca-fuzzy-set-consistency/row6/"
orig-file="https://ingorohlfing.files.wordpress.com/2016/06/row6.png"
orig-size="480,480" comments-opened="1"
image-meta="{\"aperture\":\"0\",\"credit\":\"\",\"camera\":\"\",\"caption\":\"\",\"created_timestamp\":\"0\",\"copyright\":\"\",\"focal_length\":\"0\",\"iso\":\"0\",\"shutter_speed\":\"0\",\"title\":\"\",\"orientation\":\"0\"}"
image-title="row6" image-description="" image-caption=""
medium-file="https://ingorohlfing.files.wordpress.com/2016/06/row6.png?w=300"
large-file="https://ingorohlfing.files.wordpress.com/2016/06/row6.png?w=480"
width="480" height="480"
srcset="https://ingorohlfing.files.wordpress.com/2016/06/row6.png 480w, https://ingorohlfing.files.wordpress.com/2016/06/row6.png?w=150&h=150 150w, https://ingorohlfing.files.wordpress.com/2016/06/row6.png?w=300&h=300 300w"
sizes="(max-width: 480px) 100vw, 480px"}

The intricacies of consistency values in fuzzy-set QCA are well-known
and are repeatedly a subject of debate (like here). To a lesser degree,
there is also awareness of the fact that one should watch out for
contradictions in fuzzy-set QCA. In principle, it then should not come
as a surprise that a sufficiently high consistency value masks a
contradiction, but I hadn't linked those two issues until I had a look
at this plot.

The plot conveys two points. First, it should become standard to look at
the XY-plots of all truth table rows having consistency values above the
preferred threshold. Second, in this example, the row should not be
included in the minimization process because the consistency value is
only driven upward by cases that are not members of the truth table row.
In other instances, it will be less clear cut and, say, three cases are
members of the row and consistent and one case is a contradiction. What
do you do then? In a qualitative, crisp-set view, this would be a
consistency value of 0.75 and be minimally acceptable.

I have no definite advice on how to proceed in such situations, but a
two-fold standard would be one way to go: the fuzzy-set consistency
value should be above a given threshold and the crisp-set consistency
value that only effectively relates the members of the row to each other
should be above a selected threshold. If anybody else has other ideas on
how to address this problem, I would be happy to hear about them.

 


Qualitative Comparative Analysis (QCA) is the method of choice for the analysis of set relations and has changed considerably and improved over the years.

Curiosities of QCA: Fuzzy-set consistency

Among the recent criticisms of QCA, [[Lucas and Szatrowski's (LS)
critique](http://dx.doi.org/10.1177/0081175014532763){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} stands out in multiple
respects, including its scope, its tone (QCA is "a nonanalytic means to
identify asymmetric causal illusions", p. 66), and the responses it
provoked (and the notorious mistake in their first simulation).
Naturally, the [[critical
replies](http://smx.sagepub.com/content/44/1.toc){style="color:#3366ff;"
target="_blank"} ]{style="color:#3366ff;"}focused on LS's assessment of
QCA, but I find that they make an almost equally interesting and ironic
claim at the end that went largely unnoticed.

Despite all criticism, LS attempt to add a constructive note at the end
of their article by recommending the use of process tracing and
comparative case studies instead of QCA. This is a paradoxical turn
because that is what QCA started with, as the authors themselves note.
The term 'process tracing' was barely in use when [[Ragin introduced QCA
in
1987](http://www.amazon.com/Comparative-Method-Qualitative-Quantitative-Strategies/dp/0520280032/ref=sr_1_1?s=books&ie=UTF8&qid=1427552111&sr=1-1&keywords=ragin+comparative+method#reader_0520280032){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}, although it was done in
empirical research and comparative case studies were at the center of
the debate about "[[the comparative
method](http://www.jstor.org/stable/1955513){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}".

Besides that the research goal of process tracing/case studies and QCA
differs, it is well- known that QCA started off as a middle course
between case studies and regression analysis. With respect to case
studies, QCA aimed to overcome the small-n problem in the sense of
getting the bigger picture and determining potentially redundant
conditions. We can argue as to whether or to what extent QCA succeeds in
bridging case knowledge and the formal analysis of a medium or large
number of cases. However, the argument that we can make more solid
cross-case inferences with more rather than fewer cases should be
relatively uncontroversial. At least, this is the take-home message of
Mill's own reflection about his methods and of discussions such as
[[Lieberson's](http://dx.doi.org/10.1093/sf/70.2.307){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} to which I subscribe.

In this perspective, it is ironic to plea for a return to case studies
because this brings us back to the method and associated problems that
QCA took off with. This is not to say that, as it stands, QCA works
fine. However, abandoning QCA because of some [[problems that should not
come as a
surprise](https://wordpress.com/post/6770565/754/){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} overstretches the point. Based
on [[proper
simulations](http://dx.doi.org/10.1007/s11135-015-0251-8){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} and formal assessments of QCA,
the way forward should be to improve QCA and to determine the conditions
under which it works better or worse. The use of QCA and case
studies/process tracing is not an either-or issue because there is a
proper place for both methods in the social sciences.


Among the recent criticisms of QCA, Lucas and Szatrowski’s (LS) critique stands out in multiple respects, including its scope, its tone (QCA is “a nonanalytic means to identify asymmetric causal illusions”, p. 66), and the responses it provoked (and the notorious mistake in their first simulation). Naturally, the critical replies focused on LS’s assessment of QCA, but I find that they make an almost equally interesting and ironic claim at the

The irony of Lucas/Szatrowski’s QCA criticism

How to enhance reproducibility when using fs/QCA software

The ability to reproduce published empirical studies is [central to
progress in
science](http://gking.harvard.edu/files/replication.pdf){target="_blank"}.
Reproducibility requires the proper storage and sharing of data, which
[cannot be taken for
granted](https://orgtheory.wordpress.com/2015/08/11/sociologists-need-to-be-better-at-replication-a-guest-post-by-cristobal-young/){target="_blank"},
and a detailed, step-by-step description of the empirical analysis.
Statistical research has no problems with the latter, in principle,
because a script can be compiled (do-file etc.) that can be easily
shared.

The invention of the [R package for
QCA](https://cran.r-project.org/web/packages/QCA/index.html){target="_blank"}
allows QCA users to do the same which is one major advantage of this
package. There are no numbers on the relative use of the package versus
the established [fs/QCA software offered by
Ragin](http://www.socsci.uci.edu/~cragin/fsQCA/software.shtml){target="_blank"},
but my hunch is that fs/QCA is still the dominant software.

Fs/QCA does not allow one to write a script because it is based on a
graphical user interface (GUI), but it nevertheless allows users to
ensure reproducibility, a feature that is largely ignored (based on my
knowledge of the applied QCA literature). Everything you do with your
data is documented in the results window, the window which pops up first
when you open the software, and can be exported via *File* -\> *Save*
-\> *Output*. The file type is .out, but out-files can be edited with
ordinary text editors.

This is basically what you need to document what you did. The only
challenge is to produce a file that exactly captures the analysis. Most
probably, the more cumbersome method is to try to run the analysis from
beginning to end in exactly the same way as is outlined in the
manuscript. Once you forget a step, take one that is unnecessary or
implement it incorrectly, you have to rerun the analysis from scratch.
In the second variant, you simply make sure that all steps required for
reproduction are contained in the out-file and then utilize any text
editor to remove unnecessary entries. Here, you run the risk of
producing an edited file that is missing key information for
reproduction.

In either case, the fact that fs/QCA is GUI-based should not stop one
from properly documenting the empirical analysis. For the sake of living
up to the standards of the social sciences, we hopefully will see more
such documentations of QCA research in the future.


The ability to reproduce published empirical studies is central to progress in science. Reproducibility requires the proper storage and sharing of data, which cannot be taken for granted, and a detailed, step-by-step description of the empirical analysis. Statistical research has no problems with the latter, in principle, because a script can be compiled (do-file etc.) that can be easily shared.

Messaggi di Rogue Scholar

The COMPASSS statement and QCA solution types

Don’t conflate fuzzy set membership with cases in QCA

Curiosities of QCA: Fuzzy-set consistency

The irony of Lucas/Szatrowski’s QCA criticism

Reproducible QCA studies with fs/QCA