The debate between Yann LeCun and Elon Musk, reported in Nature, questions whether science necessitates publishing results. My position is that science depends on how you produce your knowledge, not necessarily requiring publication. Only if you face the public, you need to publish about your work.

Ingo Rohlfing

Nature reports about a debate on X that started between Yann LeCun and Elon Musk. In short, the question is whether you can only call ‘science’ science when you publish your results. The Nature article is paywalled, but you can, as of now, still read the original post by LeCun that initiated the discussion.

Is it science when you don’t publish it?

Is Open Science passé?

is the question asked by Xenia Schmalz in this blogpost. I recommend reading it before I share brief thoughts on some points that are raised. I wish an open science movement was not needed anymore, but I agree this is most likely not the answer to the leading question. Neither has the open science movement failed; progress toward more transparent and credible science is simply slow.

Open Science is neither passé nor is it there yet

The LSE Impact blog has a post from May 2021 raising some reservations about the idea of ‘Slow Science’. The ‘Slow Science’ idea hasn’t really picked up in academia, as far as I can tell. The post presents some good thoughts about why the “slowness-idea” is problematic in general. I agree that slowness is not a value in itself. Sometimes, developments and events like a pandemic demand it to do research faster than one would do it otherwise.

Science should be as fast as possible and as slow as necessary to do good research

This post summarizes some (late) thoughts on the short article

 The data revolution in social science needs qualitative research

by Grigoropoulou and Small, published in Nature Human Behavior. This is an excellent article that systemizes the ways in which qualitative research should complement big data/computational social science (CSS) and gives example of work that has done this already (I understand big data/CSS to be the focus here).

Qualitative Methods in Computational Social Science

Some ideas about how the peer review process in academic publishing can be used as a device for preregistering parts of an empirical analysis.

Preregistration of an analysis requires the prior unobservability of data. One publicly declares what one intends to do before one can access data. One goal of preregistration is to diffuse potential concerns that a publication misrepresents the order in which an empirical analysis was implemented.

The peer review process as a preregistration device

Why the filling of gaps in the research literature is similar to replication and probably better than its reputation.

A common justification of empirical research is that a gap in the literature needs to be filled.

Gap-filling, puzzle-solving and replication in empirical research

On September 27, the German Science Foundation (DFG) announced its decision to award the status of a research cluster of excellence (Exzellenzcluster) to 57 cluster proposals from all disciplines.

Correlation vs causation: The case of competitive funding and research quality

About two weeks ago, COMPASSS issued a Statement on Rejecting Article Submissions because of QCA Solution Type. In short, the reasoning was that methodological work on QCA is developing and that reviewers and editors should not judge empirical work based on whether one particular solution type is interpreted as causal.

The COMPASSS statement and QCA solution types

The charge that Political Science (or other non-STEM disciplines) is lacking relevance and does not produce interesting research is made then and again, with two new pieces published these days. One is written by a political economist, stating that most research is boring;

The relevance of Political Science: Some thoughts on the recent critique

For what it’s worth, I wanted to see how often ‘process tracing’ is mentioned in the offerings on Google Books.

The rise of process tracing (in Google Books)

For some time now, a discussion has been raging about the pros and cons of set theory and the use of set-theoretic methods (STM) in the social sciences (e.g., in Sociological Methodology and the APSA Newsletter). Following up on a critical discussion by Paine and a constructive, comparative discussion of STM and regression analysis by Thiem, Baumgartner and Bol (TBB),

 Comparative Political Studies

organized a symposium on STM.

Another mistaken criticism of set theory and set-theoretic methods

Recently, I came across an interesting study from 2009, the content of which is not relevant here.

Two proposals for incentivizing data archiving

Let’s imagine you are attending a presentation and the presenter does not follow the standard intro-literature-theory-design-results-conclusion structure. Instead, she only presents the results of her research. What would be the first question you would ask? I assume at least 90 percent of the audience would ask about how she arrived at her findings.

Different tools, shared standards: The debate about DA-RT

At the end of last week, a two-day conference,

 Qualitative Comparative Analysis – Social Science Applications and Methodological Challenges

, took place in Tilburg, the Netherlands. Needless to say, the recent and ongoing wave of criticism of QCA was a key topic on the agenda and in discussions among participants.

QCA is developing… apart?

Open access and gated publishing have been discussed at length by researchers, publishers, and the media. Still, the debate got more intense when Elsevier sent takedown notices to Academia.edu. Yesterday, it got even more intense when it became public that the University of Calgary is also confronted with takedown notices from Elsevier. Elsevier responded to the critiques, but it is unlikely that this will take the heat out of the debate.

You may not like what Elsevier is doing, but…

Schlimm? 470000 entscheiden über die Koalition

Der Mitgliederentscheid wurde kritisiert, weil etwa 470000 SPD-Mitglieder über die große Koalition entscheiden.

Mit dem Quatsch aufhören III: Parteien entscheiden in eigener Sache

Der Mitgliederentscheid der SPD über den Koalitionsvertrag ist angelaufen. Je mehr Zeit seit der Ankündigung verstrichen ist, umso stärker und wilder scheint die Kritik zu werden, so dass ich nun doch einen Blogeintrag dazu schreibe (was eh wieder Zeit wurde). Aufgrund der Anhäufung von Kritik werde ich die über die nächsten Tage mehrere Einträge daraus machen.

 Alle Staatsgewalt geht vom Volke aus.

Mit dem Quatsch aufhören: Über die andauernde Kritik am SPD-Mitgliederentscheid und der Großen Koalition

Under the rubric of “Reader’s view”, I will post short reviews of books, mostly from the field of social science methods.

Reader’s view: Goertz/Mahoney (2012): A Tale of Two Cultures

The bi-annual publication of the APSA Section on Qualitative Methods and Multi-Method Research has a highly interesting and controversial symposium on confirmation bias in process tracing in its current issue.

Confirmation bias in causal qualitative research

How to properly interpret consistency and coverage values in fuzzy-set QCA.

In 2000, Ragin added the idea of fuzzy sets to Qualitative Comparative Analysis (QCA) and they have been widely used since then. However, one sometimes still finds misunderstandings about what fuzzy sets are, in particular when it comes to the interpretation of consistency and coverage scores. In a criticism of QCA by Stockemer (which is criticized for a variety of reasons and to which Stockemer again replied), he runs a fuzzy-set analysis.

Don’t conflate fuzzy set membership with cases in QCA

Alt-Bundespräsident Roman Herzog wird von Focus mit den Worten zitiert: „Im Prinzip ist die Fünf-Prozent-Hürde nicht mehr zeitgemäß. Eigentlich müssten wir die Hürde nach oben setzen.“ Die Begründung: Vor dem Hintergrund von immer mehr kleineren Parteien werde der Bundeskanzler/die Bundeskanzlerin ansonsten „nicht mehr von einer großen Mehrheit der Bevölkerung getragen“. Dies sieht Herzog als eine Gefährdung der parlamentarischen Demokratie.

Roman Herzog und die Hürde

Research on causal mechanisms is a growth industry, with the largest percentage of studies falling into the camp of qualitative research. The reason for this is the admonition that correlation is not causation, implying the claim that valid causal inference requires evidence on causal mechanisms.

Mechanisms in qualitative and quantitative research

In

 Finding Pathways: Mixed-Method Research for Studying Causal Mechanisms

, Weller and Barnes seek to explain “how the small-N component of multi-method research can meaningfully contribute and add value to the study of causal mechanisms” (quote from blurb). The book contains nine chapters, including the introduction and concluding chapter.

Review of “Finding Pathways”, by Weller/Barnes

Live and let live, or: unity and diversity in the social sciences

As an approach to and method for the analysis of set relations,

 Qualitative Comparative Analysis

(QCA) belongs in the toolbox of many social scientists (and organizational and management researchers alike). Although it has proven to be very popular, a question that one still often confronts is: what is QCA? Even those who understand the technicalities of QCA wonder what the qualitative part in QCA represents in practice.

What is QCA? Simple question, many answers

As is well known, QCA has been under intense scrutiny in recent years and subject to criticism (sometimes quite strong). I am not going to review studies on the validity of QCA that entail criticism;

Surprised, anyone? Putting the debate about QCA into context

Qualitative Comparative Analysis (QCA) is a method utilized by different disciplines in the social sciences and beyond, e.g., business economics and management. However, QCA users must still justify their choice of method more frequently than the users of other methods. Whatever the reason, it is actually not a bad thing to reflect upon the choice of a method because it should be suitable for answering our research question.

How (not) to justify running a QCA

If this was a blog post about

 the

#APSA2014, I would have to write about Friday night’s fire emergency at the Marriott (i.e., #APSAonfire) as the non-academic event that left a definite imprint (and affected me as one of the many people who had a room at the Marriott). But uncomfortable as that night had been, I will focus instead on the ongoing debate about set theory and Qualitative Comparative Analysis (QCA) which are

 the

Impressions from the APSA 2014, or: QCA under fire?

People familiar with the development of qualitative methods know that process tracing has developed rapidly over the last years. As the discussion about a method becomes broader and deeper, it becomes more important to systematize and sort the field in order to understand what it can and cannot achieve.

Three varieties of process tracing? It’s four

The choice of cases for empirical analysis is a central topic in the methods literature. The argument that quantitative and qualitative research studies different templates, which is probably most forcefully described in

 A Tale of Two Cultures

, does not come as a surprise. What I want to focus on here are the arguments on case selection in qualitative research.

Theory and substance in case selection, or: Give random selection a chance

The methods literature on process tracing took a Bayesian turn in recent years. Bayesian inference, whereby you condition on evidence in order to update your prior confidence in a hypothesis, is presented as a mode of inference one should follow in process tracing (Bennett, Andrew (2008): Process-Tracing: A Bayesian Perspective. Box-Steffensmeier, Janet M., Henry Brady and David Collier (ed.):

Frequentist inference and process tracing: an oxymoron?

Bekanntermaßen wurde der SPD-Mitgliederentscheid von einigen Verfassungsrechtlern in Frage gestellt. Mittlerweile hat das Verfassungsgericht in seiner Ablehnung eines Eilantrags auch deutlich gemacht, dass der Entscheid verfassungsrechtlich nicht zu beanstanden ist. Sigmar Gabriel hat die Verfassungsfrage des Mitgliederentscheid forsch beiseite gewischt, allerdings ist die Begründung irritierend (zumindest die, die er bei 3 nach 9 gegeben hat).

Die Verfassung und der Mitgliederentscheid: Wo Sigmar Gabriel irrt

Qualitative Methods (i.e., process tracing, set theoretic methods, informal Bayesian inference etc.) and multi-method research, in particular the combination of regression analysis or QCA with case studies, are certainly a growth industry in political science and sociology. In light of some methods panels held at the APSA Annual Meeting in Chicago and the ECPR General Conference in Bordeaux, this might actually come as a surprise.

Qualitative Methods: A Field in Search of Common Ground

Recently, Frontal 21 – a German newscast – reported about the handling of error in treatments under German health care law (about which I am not an expert). A lawyer specializing in health care law explained that it is very difficult to win lawsuits against medical doctors (MD’s) because the patient claiming to experience treatment error has to substantiate this claim (except for grave treatment error). Particularly problematic are cases where

Error in medical treatment and causation in German health care law

The current APSA Comparative Politics Newsletter is dedicated to “Doing Comparative Politics Elsewhere” (i.e. , outside of the US). Thomas Plümper contributes a discussion on Comparative Politics in Europe. In brief, Plümper argues that, until recently, the field of Comparative Politics (CP) has been dominated by qualitative methods.

Comparative politics and the choice of methods

Sourcely, an AI company, promises to streamline research by finding, summarizing, and adding credible sources in minutes. While this sounds appealing, skepticism arises as using such a tool may prioritize citing over genuine research. Initial tests revealed limited functionality, leaving doubts about its practical value in the research process.

I came across the AI company Sourcely these days. It promises to “

 Finish Your Research in Minutes. Save Your Sleep. Paste your essay to find, summarize, and add credible sources. (That’s something Google Scholar can’t do!)

” Saving time for sleep is good and if it would work, it would outperform GS, or any traditional search for citations and references, for that matter. However, I have some thoughts and concerns.

Citation-infusion of research papers with AI?

The article “Resolving empirical controversies with mechanistic evidence” discusses the potential of using evidence about mechanisms to resolve statistical disagreements and aid in choosing the correct quantitative model. While there are challenges and uncertainties in this approach, it emphasizes the value of theorizing about mechanisms and collecting evidence about them, especially in disciplines like economics.

Recently, I came across the open access article Resolving empirical controversies with mechanistic evidence. I have some thoughts on the arguments made in the article, but first a disclaimer: This is not meant to be a disciplinary beauty contest. The article raises a valuable point: Two or more quantitative models may produce contradictory results.

“Resolving empirical controversies with mechanistic evidence” – Some thoughts

In place of a generic blog post, I am reposting a short Twitter thread here. The thread is a response to an opinion piece on the Times Higher Education website titled Pay researchers for results, not plans. (Posts on the THE website require registration of an account that includes a couple of free reads.)  I copy-paste thread into this post. If you prefer to read it on Threadreader, you find it here.

Is payment for publications and paying for citations a better to way to fund research? Most likely not

When we use the Quine-McCluskey algorithm to derive a QCA solution, we can choose between the conservative, intermediate or parsimonious solution. While I do not have any figures about which solution has been produced how frequently in empirical research, it is safe to say that the conservative solution is quite popular.

The conservative QCA solution isn’t based on counterfactuals? Not so fast

When making causal (or descriptive) inferences, it is important to think about the context within which the causal relationship is expected to hold because it probably does not hold universally and, possibly, only in a limited setting. Falleti and Lynch have written an excellent article about “context” (I drop the ““ now) in relation with causal mechanisms.

“Context” is important, but (almost) useless if used as a causal category

One of the recent big and, in my view, underappreciated innovations in the field of Qualitative Comparative Analysis (QCA) is Baumgartner’s formulation of the Coincidence Analysis algorithm (CNA). Baumgartner presents it as an alternative to QCA, which I do not find convincing because I do not see QCA married to a specific algorithm.

You are a regularity theorist when using the Coincidence Analysis algorithm in QCA

More often than one might expect, television series and films offer excellent illustrations of methodological and methods-related arguments (which is worth a blog post of its own). When I was working on my paper on comparative hypothesis testing in process tracing, I was watching the first season of the terrific TV series, Homeland.

What Homeland has to do with causal inference in process tracing

Continuing the chapter-by-chapter review of Seawright’s book on Multi-Method Social Science took me longer than I imagined and it should have, but here we go again.

Review of ‘Multi-Method Social Science’ (Seawright, CUP) –  Chapter 2: Causation as A Shared Standard

Review of the first chapter of the book “Multi-Method Social Science” by Jason Seawright, published with Cambridge University Press in 2016.

Over the next weeks, I will give a chapter-by-chapter review of the new book Multi-Method Social Science: Combining Qualitative and Quantitative Tools by Jason Seawright. To my knowledge, it is the first book on nested analysis/multimethod research in the social sciences that is not an edited volume.

Review of “Multi-Method Social Science” (Seawright, CUP): Chapter 1

Das 2014er Gutachten der Expertenkommission Forschung und Innovation (EFI) schlug diese Woche einige Wellen, weil es nach wie vor eine Abwanderung der besten Wissenschaftler aus Deutschland konstatiert (

 brain drain

, wenn man den Begriff mag). Nach einem Blick in das Gutachten (Abschnitt B2, ab S. 85) sind dazu einige Anmerkungen angebracht, die die Schlagzeilen in den Kontext rücken. Erstens, die Ergebnisse sind wenig überraschend.

Das EFI-Gutachten und die Ab- und Zuwanderung deutscher Wissenschaftler: Ein paar kritische Anmerkungen

Inspired by an email exchange I had with someone on theoretically non-exclusive hypotheses in Bayesian process tracing, I believed it might be useful to write down some thoughts in a blog post. It ended up as a PDF on Github because it has a minor R element and formal notation.

Theoretically non-exclusive hypotheses in Bayesian process tracing

Many APSA 2016 panels and discussions in the Section on Qualitative and Multimethod Research and the Political Methodology Section were centered on the Data Access and Research Transparency (DART) Initiative (probably worth a blog post of its own). Even panels not explicitly dedicated to DART have digressed into this topic, which includes a short exchange between Tasha Fairfield and me in a panel on causal inference in qualitative research.

People require preregistration, not methods

Qualitative Comparative Analysis (QCA) is the method of choice for the analysis of set relations and has changed considerably and improved over the years.

Curiosities of QCA: Fuzzy-set consistency

The idea of most-likely and least-likely cases dates back to Eckstein and was one of the few remaining things in qualitative research there seemed to be no disagreement about because they are considered an asset in causal analysis. In a paper that is advance access, Beach and Pedersen (BP) now argue that process tracing and the analysis of mechanisms does not make sense with most-likely and least-likely cases.

Process tracing is possible with most-likely and least-likely cases

Among the recent criticisms of QCA, Lucas and Szatrowski’s (LS) critique stands out in multiple respects, including its scope, its tone (QCA is “a nonanalytic means to identify asymmetric causal illusions”, p. 66), and the responses it provoked (and the notorious mistake in their first simulation). Naturally, the critical replies focused on LS’s assessment of QCA, but I find that they make an almost equally interesting and ironic claim at the

The irony of Lucas/Szatrowski’s QCA criticism

How to enhance reproducibility when using fs/QCA software

The ability to reproduce published empirical studies is central to progress in science. Reproducibility requires the proper storage and sharing of data, which cannot be taken for granted, and a detailed, step-by-step description of the empirical analysis. Statistical research has no problems with the latter, in principle, because a script can be compiled (do-file etc.) that can be easily shared.

Reproducible QCA studies with fs/QCA

The social science literature is full of discussions about causation and what the best method for causal inference might be. However, a relatively small percentage of them draw on the philosophical debate about causation. Certainly, there is a great deal of talk about philosophy of science on the ontological and metaphysical level, including, for example, engagement with the relation between neo-positivism and realism.

Review of “Causation: A Very Short Introduction” by Mumford/Anjum

Most multi-method research (MMR) studies with which I am familiar start with regression analysis (or, in recent years, QCA) and perform the case studies afterward. This is the order recommended by Lieberman in his nested analysis article which, in turn, most probably reflects the widely held view that case studies are only worth doing in MMR when something is going on at the cross-case level.

What do we learn about case studies from follow-up regression analysis?

In einem Kommentar auf SPIEGEL ONLINE wird unter anderem der Mitgliederentscheid dafür verantwortlich gemacht, dass der Koalitionsvertrag wenig zukunftsorientiert sei und das Durchsetzen von Reformen zu Beginn der Legislaturperiode verhindere. Führt die Ausübung der innerparteilichen Demokratie also zu einer mutlosen Politik im hier und jetzt? Aller Wahrscheinlichkeit nach nicht.

Verhindert der Mitgliederentscheid zukunftsorientierte Politik?

Das Zögern und Zaudern der SPD vor, während und nach den Koalitionsverhandlungen wird gerne beiseite gewischt mit dem Hinweis, „der Wähler“ (und die Wählerin) wolle die große Koalition.

Mit dem Quatsch aufhören (II): “Der Wähler” will die Große Koalition

At the end of last week, I came across a blog entry by Olaf Storbeck, reporting that a rising star from German business economics, Prof. Ulrich Lichtenthaler, is faced with numerous inquiries concerning his publishing record. Two journals have already retracted three of his articles and additional articles are under scrutiny.

Quality and quantity in publishing

In the case study literature, one new pair of terms has been established by the field that concerns the distinction between

 data set observations

and

 causal process observations

. In short, Collier, Brady, and Seawright (2004: Sources of Leverage in Causal Inference: Toward an Alternative View of Methodology. Brady, Henry E. and David Collier (eds.):

 Rethinking Social Inquiry.

Don’t call it causal when it is an observation

Politics, Science, Political Science

*Is Open Science passé?* is the question asked by Xenia Schmalz [in this
blogpost](https://xeniaschmalz.blogspot.com/2024/04/is-open-science-passe.html).
I recommend reading it before I share brief thoughts on some points that
are raised.

I wish an open science movement was not needed anymore, but I agree this
is most likely not the answer to the leading question. Neither has the
open science movement failed; progress toward more transparent and
credible science is simply slow. When one looks at [how many articles
share code and
data](https://replicationnetwork.com/2024/04/10/aoi-mass-reproducibility-and-replicability-a-new-hope-by-brodeur-et-al-2024/)
(or, maybe, how many do not); [what the quality of the shared material
is](https://doi.org/10.1038/s41597-022-01143-6) (README file, code
annotation etc.); how often [substantive significance is not discussed
or misinterpreted](https://doi.org/10.1093/esr/jcw047) in quantitative
research etc., I feel there is still much room for improvement. The
question is what this means for open science and meta science.

Does it make sense to write another article on the misinterpretation of
the p-value in a given fieldor subfield? Probably yes, it depends on the
field. It may be less needed in a field like psychology because there
seems to be a larger degree of awareness to matters of research quality
and credibility. In other fields, probably including political science,
it may be interesting to review the interpretation of p-values and
substantive significance because fewer graduate students may have
received training on this. Do we need another many-analysts study
showing that results differ across analysts? I am less sure about that,
but this is a topic for a separate blog post.  

I think the reviewer comment that motivates the blog post by Xenia
Schmaltz illustrates that for many papers the study and results may be
obvious to a researcher from the field, leading to a comment like
"*What's the news here?*". For many other researchers, however, the
study may be interesting and important to read because they do not
follow closely work on open science and meta science. It is fine when an
empirical researchers pays most attention to theories and empirical
issues that are pertinent to the own subfield because this is where the
primary interest lies. For open science and meta science reserchers,
this means that there is still much work to do, including research,
teaching and publishing on topics that may seem like old news to experts
from these two fields.


Preregistration of an analysis requires the prior unobservability of
data. One publicly declares what one intends to do before one can access
data. One goal of preregistration is to [diffuse potential concerns that
a publication misrepresents the order in which an empirical analysis was
implemented](http://cega.berkeley.edu/assets/cega_events/45/HumphreysFishingPolAnal.pdf){target="_blank"
rel="noreferrer noopener"}. Reasons for prior unobservability of data
could be that the data has yet to be produced (as in experiments and
[some observational
designs](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2731920){target="_blank"
rel="noreferrer noopener"}), or that access to the data is restricted
(such as sensitive microdata that can only be accessed locally and by
certified approval of the data host). Based on a peer review experience
a friend of mine and I had, it occurred to me that there is a second way
of preregistration that works *regardless* of the prior observability of
data.

In our peer review process, a reviewer suggested to additionally test
our argument with different data than we had used. We had not thought
about using the data before and did it at the reviewer's suggestion. (I
don't remember whether we had known it; I don't think so.) The good news
for us was that an analysis with the alternative data produced evidence
that confirmed our original results. This confirmation is qualitatively
different from the results that we reported in the first submission and
it should count more, in my opinion.

The reason is that the peer review process and the reviewer comments in
particular worked as a preregistration device. The data the reviewer
recommended to use was available to us, but we weren't aware of it and
if we had been, we did not use it. What was unavailable to us was the
idea of using the data in the way proposed by the reviewer. In analogue
to the idea of 'prior unobservability of data' that underlies standard
preregistration, one could call the experience that we made *prior
unavailability of an idea* (or *prior lack of imagination* or so).

The advantage of preregistration through peer review is that it is
available regardless of the type of design -- experimental or
observational -- and the type of method -- quantitative or qualitative.
For example, imagine a researcher has done a qualitative content
analysis of publicly available speeches of party leaders. The original
analysis cannot be preregistered because the speeches have been publicly
available from the start of the project. Now, the peer review process
would work as a preregistration device if a reviewer (or an editor) has
an idea for a coding category that the researcher had not thought of.
When the researcher goes back to the speeches and codes them based on
the new category, this part of the analysis would be preregistered. The
peer review process allows one to timestamp the origin of the idea in
terms of the time when it was proposed and the person who proposed it
(the originator).

There are several requirements that have to be met for effective
preregistration through peer review, where 'effective' means that the
researcher's degrees of freedom in misrepresenting an analysis are
minimized. First, the peer review process needs to be made
*transparent*. The reader of an article needs to know who the originator
of an idea is -- the author or a reviewer -- and when the idea was
introduced by the originator. A published manuscript would need to
specify that an element of the empirical analysis was proposed by a
reviewer and that this part of the analysis was only implemented after
the reviewer proposed to do it.

Second, this information would not be enough because the reader needs to
know *how specific* the reviewer's proposal was. The more specific the
reviewer's comment, the smaller the researcher's degrees of freedom when
implementing it and the fewer opportunities to misrepresent the analysis
and "manufacture" results such that they support a preferred theoretical
interpretation.

Third, the reader needs to know what a reviewer proposed to do and what
an author *declined* to implement. With this information, the reader
would not be able to tell whether an author refused to follow the
reviewer's recommendation because of the belief that it is superfluous
or flawed in terms of substance, theory or method; or because the author
did not want to include it because the proposed analysis did produce
results that challenge the conclusion in the manuscript and were
withheld as a consequence of this. Still, it is useful to make
transparent where reviewers and authors disagreed; keeping in mind that
even many declined reviewer requests are not a strong indication of
selective reporting because the reviews could make ill guided
recommendations. The degree of disagreement would be an indication of
how selective the author was in following the reviewer's comments and
could be a guide to a follow-up technical replication of the analysis.

Altogether, this shows that preregistration through the peer review
process requires transparent peer review that makes public all versions
of the manuscript -- the first submission plus all revisions -- and all
reviewer reports, which could be anonymized for preregistration
purposes, and editorial letters.


A common justification of empirical research is that a gap in the
literature needs to be filled. I have seen this reasoning in all kinds
of papers and presentations by Master students, PhD researchers,
Postdocs and professors (probably the less, the more advanced the
academic career is). It is not wrong to justify a study with
gap-filling, but it is also does not seem to be the most exciting
argument because the natural question is: *Is the gap worth filling?* In
my limited experience, parts of the audience, at least those who ask
this question, tend to be skeptical that the question can be answered
affirmatively.

A common response to gap-filling justifications is to focus on a
different question and to solve a *puzzle*. Puzzle-solving has always
been a popular research strategy (such as
[here](https://doi.org/10.1162/002081800551127){rel="noreferrer noopener"
target="_blank"}), but my feeling is that it has grown more popular over
the last years (Day and Koivu give an [excellent teaching-focused
intro](https://doi.org/10.1080/15512169.2018.1493594){rel="noreferrer noopener"
target="_blank"}). Identiying and explaining a puzzle certainly has
merit and I do not aim to argue against it here. However, I think there
is a downside to the overemphasis of puzzle-solving and to discouring
researchers from filling gaps (to be reframed in a second as something
else).

The issue with puzzle-solving is that one has to build on existing
theoretical arguments that (supposedly) work in general, but not for all
cases that then constitute the puzzle. Theoretically, we learn something
new from puzzle-solving, which is certainly positive. However, this also
means that puzzle-solving puts a premium on *theoretical originality*
and tries to push a theory and its hypotheses forward by improving then
and explaining the anomalies.

Puzzle-solving is fine as one element in the researcher's toolbox if it
does not crowd out alternative research goals and strategies. One
alternative, which is still centered on the idea of theoretical
originality, is generic exploratory research of new empirical phenomena
such as the decarbonization of energy production.

A second alternative is g*ap-filling reframed as replication*.
Gap-filling studies are not all the same, but in my experience they are
commonly justified with the argument that a selected hypothesis hasn't
been tested yet for a specific country, or region, or institution, or
policy field, or period of time or what other dimension with variation
across cases one can think of. I do not remember a study that framed
gap-filling as a replication attempt, but the two are equivalent if
gap-filling means taking an existing argument and testing it in a
different empirical setting and with different data than the original
study.

For example, imagine a study tested a hypothesis of lobby group
influence on policy making by using cases of distributive policy making.
A gap-filling replication study could test the same hypotheses using
cases of redistributive policy-making (assuming the hypothesis is
supposed to explain lobby influence on redistributive policy-making).
The example indicates that gap-filling is not verification in the narrow
sense where one exactly repeats the original study. The gap-filling
analysis varies the type of policy compared to the original analysis,
which means that gap-filling also has an element of theoretical
originality. In comparison with puzzle-solving, it might be less
original because one "only" tests an existing argument in a different
empirical setting instead of developing the theoretical argument
further.

The reframing of gap-filling research as replication helps in clarifying
what and how much gap-filling adds to the existing state of research.
Regarding the "what"-part, replications are valuable in themselves
because no single original study is decisive and needs to be verified.
The degree to which a gap-filling study has added value depends on the
strength of evidence that has been collected in previous research. The
more existing evidence speaks for (or against) a hypothesis, the less
value I see in adding one more gap-filling/replication study. The more
ambiguous the existing evidence is and the fewer studies have been done
on a research question, the more a gap-filling study adds to our
knowledge.

Altogether, gap-filling research understood as replication is neither
interesting nor uninteresting per se. Some gaps are worth filling, while
others are not and can be left as they are. Whether a gap-filling study
is of the first type or the second can only be determined by extending
the perspective and discussing how much it adds to the existing body of
research on the same topic.


The ability to reproduce published empirical studies is [central to
progress in
science](http://gking.harvard.edu/files/replication.pdf){target="_blank"}.
Reproducibility requires the proper storage and sharing of data, which
[cannot be taken for
granted](https://orgtheory.wordpress.com/2015/08/11/sociologists-need-to-be-better-at-replication-a-guest-post-by-cristobal-young/){target="_blank"},
and a detailed, step-by-step description of the empirical analysis.
Statistical research has no problems with the latter, in principle,
because a script can be compiled (do-file etc.) that can be easily
shared.

The invention of the [R package for
QCA](https://cran.r-project.org/web/packages/QCA/index.html){target="_blank"}
allows QCA users to do the same which is one major advantage of this
package. There are no numbers on the relative use of the package versus
the established [fs/QCA software offered by
Ragin](http://www.socsci.uci.edu/~cragin/fsQCA/software.shtml){target="_blank"},
but my hunch is that fs/QCA is still the dominant software.

Fs/QCA does not allow one to write a script because it is based on a
graphical user interface (GUI), but it nevertheless allows users to
ensure reproducibility, a feature that is largely ignored (based on my
knowledge of the applied QCA literature). Everything you do with your
data is documented in the results window, the window which pops up first
when you open the software, and can be exported via *File* -\> *Save*
-\> *Output*. The file type is .out, but out-files can be edited with
ordinary text editors.

This is basically what you need to document what you did. The only
challenge is to produce a file that exactly captures the analysis. Most
probably, the more cumbersome method is to try to run the analysis from
beginning to end in exactly the same way as is outlined in the
manuscript. Once you forget a step, take one that is unnecessary or
implement it incorrectly, you have to rerun the analysis from scratch.
In the second variant, you simply make sure that all steps required for
reproduction are contained in the out-file and then utilize any text
editor to remove unnecessary entries. Here, you run the risk of
producing an edited file that is missing key information for
reproduction.

In either case, the fact that fs/QCA is GUI-based should not stop one
from properly documenting the empirical analysis. For the sake of living
up to the standards of the social sciences, we hopefully will see more
such documentations of QCA research in the future.


Recently, I came across an interesting study from 2009, the content of
which is not relevant here. However, I was not convinced by some of the
statistical analyses and decided to ask the authors whether they could
share the data with me. They replied they might have lost the data and
would have to search for it. The permanent loss of data is always
annoying, but for this study, it would be particularly harmful because
it is based on an original survey that would be difficult to replicate.
(In addition to the possibility that an analysis of newly collected data
might not replicate the results from approximately 10-15 years ago
because of time-varying effects.)

This is not an unfamiliar problem and made me think, as have many people
before, how one could increase incentives for data archiving in
repositories such as the [[Dataverse
Network](https://dataverse.harvard.edu/){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}. Since positive incentives
such as data citations do not seem to achieve this goal, two negative
incentives might be more effective. If authors do not make their data
available in a repository, are asked to share their data and cannot
deliver it for whatever reason, one could take two courses of action.

First, the article for which no data are available for reproduction
would be retracted. This might seem to be a harsh response and would put
such articles on the same level as work in which the authors had
committed plagiarism or data fabrication. However, we should ask
ourselves: who has the burden of proof showing that an empirical
analysis was carried out and needs to ensure that it can be reproduced?
The burden should rest with the authors and if they can't deliver by
sharing data and code, the study *looks* similar to one that is based on
fabricated results that lack any preceding empirical work. I am not
saying all studies for which no reproduction data are available have
been fabricated, but, empirically, we cannot tell the difference. So one
might wonder why they should be treated differently.

A second, less far-reaching measure would be to flag publications with
something like "Data not available. Study cannot be reproduced." Such a
stamp should be put on the publisher's website and on the publication
itself (e.g., on every page of a PDF). The publication would not be
retracted, but the reader would be provided with the important auxiliary
information that the results she is looking at cannot be reproduced. No
single study can deliver conclusive results and all should be
interpreted with caution and this should be even more the case when the
underlying data is nonexistent.

The best option, of course, remains data archiving immediately following
publication of a study and, hopefully, more and more journals and book
publishers will make the online depositing of data and code a necessary
requirement for the acceptance of a publication. Until we have reached
this point, negative incentives might be useful complementary
instruments. Evidently, the two proposals also require the collaboration
of publishers and the retraction of non-reproducible articles might be
difficult to achieve. However, flagging publications as lacking data for
reproduction might be feasible once the loss of data is made public and
brought to the editors' attention, who hopefully, would feel compelled
to take at least some form of action.
