Ingo Rohlfing

Many APSA 2016 panels and discussions in the Section on Qualitative and
Multimethod Research and the Political Methodology Section were centered
on the [Data Access and Research Transparency (DART)
Initiative](https://dialogueondart.org/){target="_blank"} (probably
worth a blog post of its own). Even panels not explicitly dedicated to
DART have digressed into this topic, which includes a short exchange
between [Tasha
Fairfield](http://www.lse.ac.uk/internationalDevelopment/people/fairfieldT.aspx){target="_blank"}
and me in a panel on causal inference in qualitative research.

**Preregistered Bayesian process tracing?  
**We discussed whether there is a need for preregistration if you do
Bayesian process tracing. Admittedly, this seems to be a topic that is
very remote from the actual process tracing research being done at the
moment. However, a couple of panels at this and previous APSA meetings
have shown that Bayesian process tracing is one of the emerging topics,
if not *the* emerging topic currently taught by different people at
different venues. It therefore seems a reasonable guess that we will see
it in published process tracing studies over the next years.

Compared to Bayesian process tracing, preregistration seems to be an
even more remote topic. Again, however, there is a "however" because it
is an important issue in quantitative research across different fields
(sociology, psychology, economics, [political
science](http://pan.oxfordjournals.org/content/21/1.toc){target="_blank"})
and there are discussions of whether it is also needed and [applicable
to qualitative research](http://dx.doi.org/10.1017/S1049096515000189).

**Needed or not?  
**On the "whether" side, Fairfield and I disagreed in principle on
whether we need preregistration for Bayesian process tracing. I say
"yes" because there are several research design decisions that could
bias your study in the direction of making a positive finding. One
obvious point is the priors of the hypotheses you want to test. If you
gather mostly weak evidence in process tracing, meaning the likelihood
ratio is not much different from 1 for most pieces of evidence, then you
might want to fix the prior of your favorite hypothesis a little bit
higher than you initially intended because this gives you a higher
posterior.

Another point is the decision as to when to stop collecting evidence.
You might plan to stay in an archive for, say, a week, but then decide
to go home after three days because you've found a plethora of strong
evidence in favor of your hypothesis and might not want to waste
resources on four more days. This might seem like a reasonable decision,
but one that differs from your initial plan. For me, this is worth
making transparent because your practical stopping rule is made
conditional on the quality of the evidence and you leave behind many
sources that you may have meant to analyze before having seen some
sources during the first three days. (You might ask yourself whether you
would also go home if you mostly find strong evidence *against* your
hypothesis during the first three days. If you say "no", you see what
the issue is here.)

Fairfield disagreed with me based on her belief in the numerous
advantages of Bayesian inference (some of which I share and some, I
don't). I am afraid I might not be able to exactly reproduce their
arguments with a time lag of about four weeks, but her arguments against
preregistering priors seem two-fold. First, priors "wash out" as you
accumulate evidence, which I do not like as an argument because it
downplays the importance of the priors (and is not necessarily true, I'd
say). Second, Bayesian inference is kind of an evidence-processing
machine that is continuously running; it breaks down the distinction
between induction and deduction and renders it unnecessary to fix priors
at some point of your research at a specific level. On the question of
the use of sources and the practical stopping rule, Fairfield's view is
that you just process the evidence that you have and the posteriors then
are what they are.

**Practice vs principles  
**It became clear to me only after the panel that we were talking on two
different dimensions. I was arguing for preregistration because
researchers are fallible; Fairfield argued against preregistration based
on her belief in the inherent benefits of Bayesian inference and process
tracing. Regardless of how valid my two specific points are, it holds
that no method requires preregistration because the misuse of a method
is not inherent to it. *p*-hacking is not a problem of frequentist
statistics, but its misuse by researchers (you might believe p-values do
not answer meaningful research questions, but this is unrelated to
preregistration).

The same holds true for Bayesian process tracing. In principle, there
are no problems and the method works flawlessly. In practice, it is done
by researchers seeking a PhD or tenure,  wanting to get published, or
not understanding why decisions they make bias their results in one
direction or the other. This includes the formulation of priors, the
processing of sources and the interpretation of evidence.

**Odysseus and the sirens  
**Consider yourself being Odysseus on the way to do proper research, but
you are subject to the sirens' call of publishing. You are tempted to
give in and it does not matter what boat you are on because any boat can
be directed toward the sirens. You can avoid this by tying yourself to
the mast and preregistering your research design before you set sail.

To some degree, the Odysseus analogy fails because preregistration does
not mean that you really restrain yourself. You just write down your
research design and plan before you do the empirical analysis; there
might be reasons to deviate from the preregistered decisions and you can
do so, but it then becomes transparent what you intended to do and what
you actually did for what reason. With regard to Odysseus, this means
you could steer your boat to the sirens, but at least you cannot claim
that you wanted to go there all along.


Many APSA 2016 panels and discussions in the Section on Qualitative and Multimethod Research and the Political Methodology Section were centered on the Data Access and Research Transparency (DART) Initiative (probably worth a blog post of its own). Even panels not explicitly dedicated to DART have digressed into this topic, which includes a short exchange between Tasha Fairfield and me in a panel on causal inference in qualitative research.

People require preregistration, not methods

Let's imagine you are attending a presentation and the presenter does
not follow the standard
intro-literature-theory-design-results-conclusion structure. Instead,
she only presents the results of her research. What would be the first
question you would ask? I assume at least 90 percent of the audience
would ask about how she arrived at her findings. This hypothetical
example puts us right in the middle of the idea behind the [[Data Access
and Research Transparency Initiative
(DA-RT)](http://www.dartstatement.org/){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}.

**The RT in DA-RT**

One might disagree with the view that [[science is all about method
rather than
substance](https://books.google.de/books?id=A7VFF-JR3b8C&lpg=PP1&dq=designing%20social%20inquiry%20scientific%20inference%20in%20qualitative%20research&hl=de&pg=PA9#v=onepage&q&f=false){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}, but no one can reasonably
claim that method and design issues are irrelevant to science and for
determining what we can make of new findings of empirical research. It
is therefore safe to argue that transparency about every step of
analysis potentially influencing what we find and conclude in empirical
research is mandatory, accounting for the "RT" in DA-RT.

**The DA in DA-RT**

If we want to determine if the results are robust to different design
and modeling decisions, we also need access to the data, accounting for
the "DA". As [[Rick Wilson pointed
out](https://rkwrice.wordpress.com/2015/11/10/da-rt-top-and-rolling-back-transparency/){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}, this concerns access to the
raw data and not only to the final, processed dataset. Data access is
also crucial to [[cross-checking for mistakes in data preparation and
data
analysis](https://politicalsciencereplication.wordpress.com/2015/11/07/political-scientists-trying-to-delay-research-transparency/){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}; we all want to avoid
mistakes, but they can happen and, potentially, making data available
can be considered a means of crowdsourcing the task of checking for
errors. Beyond others scrutinizing our work, easy data access and
transparency enhance replication attempts and building on existing data
and studies for the accumulation of knowledge. (In quantitative
research, one should also not forget the opportunity to learn something
from how others processed the data and the syntax they used.)

Taken together, DA-RT makes for better science because we can understand
and cross-validate what others did and build on their work. Still, with
[[editors of more than two dozen journals aiming to enforce
DA-RT](http://www.dartstatement.org/#!blank/c22sl){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} by the beginning of next year,
there is now a vibrant debate about whether DA-RT is meaningful across
methods and, if so, how.

As should be clear by now, I am sympathetic to the DA-RT initiative.
However, I also see merit in some of the arguments that are floating
around, while I find others weaker or misleading. Here is my view on
some issues that are currently being discussed.

**An implementation delay needs a firm timetable**

A [[petition to delay the implementation of
DA-RT](https://docs.google.com/forms/d/1BWFO6462XNPBO8MyxV5WAcFtWn4m0fSXuOwq84FodKM/viewform){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} has been signed by 1.173
scholars. It received
[[support](http://www.the-plot.org/2015/11/02/further-thoughts-on-da-rt/){style="color:#3366ff;"
target="_blank"} ]{style="color:#3366ff;"}and has provoked a couple of
[[counterstatements](http://www.mathofpolitics.com/2015/11/06/responding-to-a-petition-to-nobody-or-everybody/){style="color:#3366ff;"
target="_blank"} ]{style="color:#3366ff;"}and
[[criticisms](https://politicalsciencereplication.wordpress.com/2015/11/07/political-scientists-trying-to-delay-research-transparency/){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}. I believe a different framing
of that petition would have increased mutual understanding. It is all
centered on the request of a delay. Literally, delaying something
implies that whatever is delayed will eventually come about. However, we
all know that delays sometimes serve as an instrument for bringing
something to an end. My reading of some criticisms of the petition is
that they interpret it in the latter way. Such misunderstandings could
have been diffused by making clearer that the petition and its
signatories want to see DA-RT implemented, but that more discussion is
needed (e.g., perhaps titling it, "Toward a More Integrative DA-RT
Initiative").

Some might also think that there is no need for a delay because the
DA-RT Initiative [[started in
2010](http://www.dartstatement.org/#!about/c24vq){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}. I do not know whether
relevant groups were able to participate in the work on DA-RT and some
just recognized the implications of DA-RT enforcement too late. But if
DA-RT simply moves forward as it stands, it has the potential to drive a
wedge into our discipline. I do not see this as a risk worth taking and
would prefer a delay over risking that.

For this to occur, one would need to remedy a second shortcoming of the
petition, i.e., it does not lay out a roadmap. If implementation is
delayed, there should be a scheduled plan through which everyone could
submit his opinion, potentially resulting in a revised DA-RT statement.
The current DA-RT statement and situation could be the default option if
the modification fails, ensuring that some improvement takes place.

**The DA-RT idea is not driven by quantitative thinking**

Some researchers' criticism is that [[DA-RT follows a quantitative
template](http://duckofminerva.com/2015/11/put-a-da-rt-in-it.html){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"} because, for quantitative
methodologists, it is easier to ensure data access and transparency.
This might be so (but see below), but a misunderstanding is at stake
here which is not uncommon in the methods field when quantitative and
qualitative methods are involved.

The general idea behind DA-RT is not in any sense quantitative in
nature. Quantitative research *implements* DA-RT in a specific way --
making data and code available -- that cannot be transferred to the
qualitative domain, but who is demanding this? In terms of a [[famous
book
title](http://www.amazon.com/gp/product/B0087PDK84?keywords=rethinking%20social%20inquiry&qid=1447446793&ref_=sr_1_2&sr=8-2){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}, we are talking about "diverse
tools, shared standards". There is nothing quantitative about doing
science transparently, but the tools for transparency required for
process tracing, Comparative Historical Analysis, ethnography, etc.
might well differ and this is what needs to be worked out (as stated in
the delay petition).

**DA-RT means a great deal of work, regardless of the applied method.**

As an example for the implications, qualitative researchers ask whether
it would become necessary to transcribe all their interviews because
this is a lot of work (here it is pointed out that ethical issues are
[[taken care of by
DA-RT](https://politicalsciencereplication.wordpress.com/2015/11/07/political-scientists-trying-to-delay-research-transparency/){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}). The answer should be, in
principle, "yes". Other researchers should be able to assess the way in
which you did your interviews, i.e., what questions were asked and when
and what was not asked. In processing the interview, did you overlook
interviewee statements that would alter the conclusions? Did you
correctly assign codes when doing a document analysis? Seen from the
other side, when you only present parts of the interviews and they are
not fully accessible, you create uncertainty about how you did your
research, which undermines confidence in the validity of your
conclusions.

Someone who has never done quantitative research might actually
underestimate the effort it takes to be transparent in quantitative
studies. You shouldn't upload some data and some syntax, but all the
data you took for building your final dataset and an annotated file with
syntax. Depending on the study, this can easily mount up to more than
1000 lines of code. Making sure that the code works and matches the
results in the publication and annotating it is also a lot of work (also
when you use Markdown, etc.).

I do not intend to launch a beauty contest between methods to decide how
hard or easy it is to achieve DA-RT when using one method or the other.
But it would help to agree that achieving DA-RT is always a lot of work,
regardless of what method you use, that failing to do the required work
weakens your research, and that achieving DA-RT is worth the effort.

**It is not about philosophy of science  
**

Some discount DA-RT because it is inspired by
[[neo-positivism](http://dx.doi.org/10.1017/S1537592715000031){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}. I might not know enough about
philosophy of science, but I do not see the bigger problem here
(concurring with [[Thomas
Leeper](http://thomasleeper.com/2015/11/reproducible-qualitative-research/){style="color:#3366ff;"
target="_blank"}]{style="color:#3366ff;"}). Giving access to data and
observations for the sake of reproducibility might be the neo-positivist
element because it presumes a duality of mind and world.

Regardless of whether my reading is correct, I believe we should not
overload the discussion with philosophical issues. Many methods,
including QCA, process tracing and Comparative Historical Analysis --
the latter two being mentioned in the delay petition --, became more
formalized and standardized in recent years. I do not see inherent
challenges in achieving DA-RT for them, be they labeled neo-positivist
or not.

For interpretivist methods, ethnography, etc., the unease might be
larger, if only because "data" might ring too quantitative in
qualitative ears. But we can easily replace "data" with "material"
without changing anything else about DA-RT (call it MA-RT, if you like).
Still, I think that researchers from ethnography can and should explain
what they did and how they did it. If they do not use software, assign
codes to parts of their material, and are not able to pinpoint
individual observations because it is incompatible with their approach,
that's fine with me. Nonetheless, thinking about transparency and the
accessibility of their analysis should also be useful for disciplining
qualitative research and the writing process. A uniform standard
matters, above all, and the tool for meeting it needs to fit the
research approach.

**Relevance and rigor is a false dichotomy**

Some argue that DA-RT requirements stifle innovation because the
[pursuit of rigorous research trumps relevant
research](http://dx.doi.org/10.1017/S1537592715000031){target="_blank"}.
I do not see relevance and rigor as an inherent trade-off that denies
relevance to methodologically rigorous research, and excuses relevant
research from pursuing rigor. I do not see any reason why, for example,
exploratory qualitative research should meet lower standards than
confirmatory research using whatever method. The value of exploratory
research lies in producing new insights, sometimes groundbreaking ones.
Still, I would like to know how interviewees were selected; what was
discussed in the interviews; how archival sources were selected and
processed, etc. There is better and worse relevant research, the
difference lying in the degree of transparency.

**How DA-RT do you go?**

Only the publication of fully transparent research, however this is
defined, represents the highest standard currently pursued. An
alternative would be to have a checklist for different methods --
quantitative, QCA, process tracing, etc. -- and tick boxes. When
interviews are not fully transcribed, the corresponding box does not get
ticked. When you only offer a processed dataset for a quantitative
analysis, the corresponding box gets ticked; however, the box
"Accessibility of raw data" goes unticked.

Information about the degree of transparency would have to be placed
right above the abstract so that everybody can see the checklist. It
would then be up to the researcher to decide how transparent she wants
to be and the reader to decide how much faith she wants to put in the
analysis. This might solve some of the concerns that access to certain
journals will be impossible if the highest DA-RT statements are not
fulfilled.

This might not be the ideal and there might be other ways to move
forward, but it should be clear that the status quo would be a step
backward. We need more transparency in political science.


Let’s imagine you are attending a presentation and the presenter does not follow the standard intro-literature-theory-design-results-conclusion structure. Instead, she only presents the results of her research. What would be the first question you would ask? I assume at least 90 percent of the audience would ask about how she arrived at her findings.

Publicaciones de Rogue Scholar

People require preregistration, not methods

Different tools, shared standards: The debate about DA-RT