Cameron Neylon

Issues surrounding the relationship of Open Research and replication
seems to be the meme of the week. [Abhishek
Tiwari](http://www.abhishek-tiwari.com/) provided [notes on a
debate](http://www.abhishek-tiwari.com/2009/07/public-access-to-data-and-models-yes-no.html)
describing concerns about how open research could damage replication and
[Sabine Hossenfelder](http://backreaction.blogspot.com) explored the
same issue in [a blog
post](http://backreaction.blogspot.com/2009/07/and-how-open-would-you-want-your.html).
The concern fundamentally is that by providing more of the details of
our research we may actually be damaging the research effort by reducing
the motivation to reproduce published findings or worse, as Sabine
suggests, encouraging group think and a lack of creative questioning.

I have to admit that even at a naive level I find this argument
peculiar. There is no question that in aiming to reproduce or confirm
experimental findings it may be helpful to carry out that process in
isolation, or with some portion of the available information withheld.
This can obviously increase the quality and power of the confirmation,
making it more general. Indeed the question of how and when to do this
most effectively is very interesting and bears some thought. The
optimization of these descisions in specific cases will be important
part of improving research quality. What I find peculiar is the apparent
belief in many quarters (but not necessarily Sabine who I think has a
much more sophisticated view) that this optimization is best encouraged
by not bothering to make information available. We can always *choose*
not to access information if it is available but if it is not we cannot
choose to look at it. Indeed to allow the optimization of the
confirmation process it is crucial that we *could* have access to the
information if we so decided.

But I think there is a deeper problem than the optimization issue. I
think that the argument also involves two category errors. Firstly we
need to distinguish between different types of confirmation. There is
pure mechanical replication, perhaps just to improve statistical power
or to re-use a technique for a different experiment. In this case you
want as much detail as possible about how the original process was
carried out because there is no point in changing the details. The whole
point of the exercise is to keep things as similar as possible. I would
suggest the use of the term "reproduction" to mean a slightly different
case. Here the process or experiment "looks the same" and is intended to
produce the same result but the details of the process are not tightly
controlled. The purpose of the exercise is to determine how robust the
result or output is to modified conditions. Here withholding, or not
using, some information could be very useful. Finally there is the
process of actually doing something quite different with the intention
of testing an *idea* or a claim from a different direction with an
entirely different experiment or process. I would refer to this as
"confirmation". The concerns of those arguing against providing detailed
information lie primarily with confirmation, but the data and process
sharing we are talking about relates more to replication and
reproduction. The main efficiency gains lie in simply re-using shared
process to get down a scientific path more rapidly rather than
situations where the process itself is the subject of the scientific
investigation.

The second category error is somewhat related in as much as the concerns
around "group-think" refer to claims and ideas whereas the objects we
are trying to encourage sharing when we talk about open research are
more likely to be tools and data. Again, it seems peculiar to argue that
the process of thinking independently about research claims is aided by
*reducingÂ * the amount of data available. There is a more subtle
argument that Sabine is making and possibly [Harry
Collins](http://en.wikipedia.org/wiki/Harry_Collins) would make a
similar one, that the expression of tools and data may be inseparable
from the ideas that drove their creation and collection. I would still
argue however that it is better to actively choose to omit information
from creative or critical thinking rather than be forced to work in the
dark. I agree that we may need to think carefully about how we can
effectively do this and I think that would be an interesting discussion
to have with people like Harry.

But the argument that we shouldn't share because it makes life "too
easy" seems dangerous to me. Taking that argument to its extreme we
should remove the methods section from papers altogether. In many cases
it feels like we already have and I have to say that in day to day
research that certainly doesn't feel helpful.

*Sabine also makes a good point, that Michael Nielsen also has from time
to time, that these discussions are very focussed on experimental and
specifically hypothesis driven research. It bears some thinking about
but I don't really know enough about theoretical research to have
anything useful to add. But it is the reason that some of the language
in this post may seem a bit tortured.*


Issues surrounding the relationship of Open Research and replication seems to be the meme of the week. Abhishek Tiwari provided notes on a debate describing concerns about how open research could damage replication and Sabine Hossenfelder explored the same issue in a blog post.

Messaggi di Rogue Scholar

Replication, reproduction, confirmation. What is the optimal mix?