Cameron Neylon

*The following is the text from which I spoke today at the
[.Astronomy](http://dotastronomy.com/) conference. I think there is some
video available on the .[Astronomy UStream
account](http://www.ustream.tv/channel/.astronomy) and I also have audio
which I will put up somewhere soon.*

Thereâ€™s a funny thing about the science and coding communities. Each
seems to think that the other has all the answers. Maybe the grass is
just greenerâ€¦For many years as an experimental scientist I looked
jealously at both computational scientists and coders in general.
Â Wouldnâ€™t it be so much less stressfull, I naively thought, to have
systems that would do what they were told, to be easily able to re-run
experiments and to be able to rely on getting the same answer. Â Above
all, I thought, imagine the convenience of just being able to take
someone elseâ€™s work and being able to easily and quickly apply it to
my own problems.

There is something of a mythology around code, and perhaps more so
around open source, that it can be relied on, that there is a toolkit
out there already for every problem. That there is a Ruby Gem, or an R
library for every problem, or most memorably that I can sit on a python
command line and just demand [antigravity by importing
it](http://xkcd.com/353/). Sometimes these things are true, but Iâ€™m
guessing that everyone has experience of it not being true. Of the
python library that looks as though it is using dictionaries but is
actually using some bizarre custom data type, the badly documented ruby
gem, or the perlâ€¦well, just the perl really. The mythology doesnâ€™t
quite live up to the hype. Or at least not as often as we might like.

But if us experimental scientists have an overoptimistic view of how
repeatable and reliable computational tools are then computer scientists
have an equally unrealistic view of how experimental science works.
[Greg Wilson](http://third-bit.com/), one of the great innovators in
computer science education once said, while criticizing documentation
and testing standards of scientific code â€œAn experimental scientist
would never get away with not providing their data, not providing their
working. Experimental science is expected to be reproducible from the
detailed methodology givenâ€¦.â€?â€¦.data providedâ€¦detailed
methodologyâ€¦reproducibleâ€¦this doesnâ€™t really sound like the
experimental science literature that I know.

[Ted Pedersen](http://www.d.umn.edu/~tpederse/) in an article with the
wonderful title â€œ[Empiricism is not a matter of
faith](http://portal.acm.org/citation.cfm?id=1450333)â€? excoriates
computational linguistics by holding it up to what he sees as the much
higher standards of reproducibility and detailed description of
methodology in experimental science. Yet Iâ€™ve never been able to
reproduce an experiment based only on a paper in my life.

What is interesting about both of these view points is that we are
projecting our very real desire to raise standards against a mythology
of someone elseâ€™s practice. There seems to be a need to view some
other communityâ€™s practice as the example rather than finding examples
within our own. This is odd because it is precisely the best examples,
within each community, that inspire the other. There are experimental
scientists that give detailed step by step instructions to enable others
to repeat their work, who make the details of the protocols available
online, and who work within their groups to the highest standards of
reproducibility that are possible in the physical world.Â  Equally there
are open source libraries and programmes with documentation that are
both succinct and detailed, that just works when you import the library,
that is fully tested and comes with everything you need to make sure it
will work with your systems. Or that breaks in an informative way,
making it clear what you need to do with your own code to get it
working.

If we think about what makes science work; effective communication,
continual testing and refinement, public criticism of claims and ideas;
the things that make up good science, and mean that I had a laptop to
write this talk on this morning, that meant the train and taxi I caught
actually run, that, more seriously a significant portion of the people
in this room did not in fact die in childhood. If we look at these
things then we see a very strong correspondence with good practice in
software development. High quality and *useful* documentation is key to
good software libraries.Â  You can be as open source as you like but if
no-one can understand your code theyâ€™re not going to use it. Controls,
positive and negative, statistical and analytical are basically unit
tests. Critique of any experimental result comes down to asking whether
each aspect of the experiment is behaving the way it should, has each
process been tested that a standard input gives the expected output. In
a very real sense experiment is an API layer we use to interact with the
underlying principles of nature.

So this is a nice analogy, but I think we can take this further, in fact
I think that code and experiment are actually linked at a deeper level.
Both are an instantiation of process that take inputs and generate
outputs. These are (to a first approximation â€" good enough for this
discussion) deterministic in any given instance. But they are
meaningless without context. Useless without the meaning that
documentation and testing provide.

Let me give you an example. [Ana Nelson](http://ananelson.com/) has
written a lovely documentation tool called [Dexy](http://dexy.it/). This
builds on concepts of [literate
programming](http://en.wikipedia.org/wiki/Literate_programming) in a
beautifully elegant and simple way. [Take a look](http://dexy.it) for
the details but in essence it enables you to directly incorporate the
results of arbitrary running code into your documentation. As you
document what your code does you provide examples, parts of the process
that are actively running, *and testing* the code as you go. If you
break the method you break your documentation. It is also not an
accident that if you are thinking about documentation as you build your
code then it helps to create good modular structures that are *easy to
understand* and therefore both easy to use and easy to communicate. They
may be a little more work to write but the value you are creating by
*thinking* about the documentation up front means you are motivated to
capture this up front. [Design by
contract](http://en.wikipedia.org/wiki/Design_by_contract) and [test
driven
development](http://en.wikipedia.org/wiki/Test-driven_development "Test-driven development"){.zem_slink
rel="wikipedia"} are tough, Documentation Driven Development can really
help drive good process.

Too often when we write a scientific paper itâ€™s the last part of the
process. We fabricate a story that makes sense so that we can fit in the
bits we want to. Now thereâ€™s nothing wrong with this. Humans are
narrative processing systems, we need stories to make sense of the
world. But its not the whole story. What if, as we collect and capture
the events that we ultimately use to tell our story, that we also
collect and structure the story of what actually happened? Of the
experiments that didnâ€™t work, of the statistical spread of good and
bad results. Thereâ€™s a sarcastic term in synthetic organic chemistry,
the â€œAmerican Yieldâ€? in which we imagine that 20 PhD students have
been tasked with making a compound and the one who manages to get the
highest overall yield gets to be first author. This isnâ€™t actually a
particularly useful number. Much more useful to the chemist who wants to
use this prep is the spread of values, information that is generally
thrown away. The difference between actually incorporating the running
of the code into the documentation, and just showing one log file, cut
and pasted, from when it worked well. You lose the information about
when it *doesnâ€™t* work.

Other tools from coding can also provide inspiration. Tools like
[Hudson](http://hudson-ci.org/) for continuous integration. Everytime
the code base is changed everything gets re-built, dependencies are
tested, unit tests run, and a record of what gets broken. If you want to
do X you do want to use this version of that library.Â  This isnâ€™t a
problem. In any large codebase things are going to get broken as changes
are made, you change something, see what is broken, then go back and
gradually fix those things until youâ€™re ready to create commit to the
main branch (at which point someone else has broken somethingâ€¦)

Science *is* continuous integration. This is what we do, we make changes
, we check what they break, see if the dependencies still hold and if
necessary go back and fix them. This is after all where the interesting
science is. Or it would be if we did it properly. [David
Shotton](http://www.imaging.ox.ac.uk/network-members/dshotton) and
others have spoken about the question of â€œcitation creepâ€? or
"hedging erosion" \[see for example [this
presentation](http://www.slideshare.net/anitawaard/hedging-fact-from-claim-science-as-mythology)
by [Anita de
Waard](http://www.elsevier.com/wps/find/newsroomhome.newsroom/bio_anitadewaard)\].
This is where something initially reported in one paper as a
possibility, or even just a speculation gets converted into fact by a
process of citation. What starts as â€œâ€¦it seems possible thatâ€¦â€?
can get turned into â€œâ€¦as we know that X causes Y (Bloggs et al,
2009)â€¦â€? within 18 months or a couple of citations. Scientists are
actually not very good at checking their dependencies. And they have a
tendency of coming back to bite us in exactly the same way as a quick
patch that wasnâ€™t properly tested can.

Just imagine if we could do this. If everytime a new paper was added to
the literature we could run a test against the rest. Check all the
dependenciesâ€¦if this isnâ€™t true then all of these other papers in
doubt as wellâ€¦indeed if we could unit test papers would it be worth
peer reviewing them? There is good evidence that pair coding works, and
little evidence that traditional peer review does. What can we learn
from this to make the QA processes in science and software development
better?

I could multiply examples. What would an agile lab look like? What would
be needed to make it work? What can successful library development
communities tell us about sharing samples, and what can the best data
repositories tell us about building the sites for sharing code? How can
we apply the lessons of [StackOverflow](http://stackoverflow.com/) to a
new generation of textbooks and how can we best package up descriptions
of experimental protocols in a way that provides the same functionality
as sharing an [Amazon Machine
Image](http://en.wikipedia.org/wiki/Amazon_Machine_Image).

Best practice in coding mirrors best practice in science. Documentation,
testing, integration are at the core. Best practice is also a long way
ahead of common practice in both science and coding. Both, perhaps are
driven increasingly by a celebrity culture that is more dependent on
what your outputs look like (and where they get published) than whether
anyone uses them. Testing and documentation are hardly glamorous
activities.

So what can we do about it? Improving practice is an arduous task. Many
people are doing good work here with training programmes, tools,
standards development and calls to action online and in the scientific
literature. Too many people and organizations for me to call out and
none of them getting the credit they deserve.

One of the things I have been involved with is to try and provide a
venue, a prestigious venue, where people can present code that has been
developed to the highest standards. [Open Research
Computation](http://www.openresearchcomputation.com/), a new Open Access
journal from [BioMedCentral](http://www.biomedcentral.com/), will
publish papers that describe software for research. Our selection
criteria donâ€™t depend on how important the research problem is, but on
the availability, documentation, and testing of the code. We expect the
examples given in these papers to be reproducible, by which we mean that
the software, the source code, the data, and the methodology are
provided and described well enough that it is possible to reproduce
those examples.Â  By applying high standards, and by working with
authors to help them reach those standards we aim to provide a venue
which is both useful *and* prestigious. Think about it, a journal that
contains papers describing the most useful and useable tools and
libraries is going to get a few citations and (whisper it) ought to get
a pretty good impact factor. *I* donâ€™t care about impact factors but I
know the reality on the ground is that that those of you looking for
jobs or trying to keep them do need to worry about them.

In the end, the problem with a journal, or with code, or with science is
that we want everyone else to provide the best documentation, the best
tested code and procedures, but its hard to justify doing it ourselves.
I mean I just need something that works; yesterday. I donâ€™t have time
to write the tests in advance, think about the architecture, re-read all
of that literature to check what it really said. Tools that make this
easier will help, tools like Dexy and Hudson, or lab notebooks that
capture what weâ€™re doing and what we are creating, rather than what we
said we would do, or what we imagine we did in retrospect.

But itâ€™s motivation that is key here. How do you motivate people do
the work up front? You can tell them that they have to of course but
really these things work best when people want to make the effort. The
rewards for making your work re-usable can be enormous but they are
usually further down the road than the moment where you make the choice
not to bother. And those rewards are less important to most people than
getting to the Nature paper, or getting mentioned in Tim Oâ€™Reillyâ€™s
tweet stream.

It has to be clear that making things re-usable is the highest
contribution that you can make, and for it to be rewarded accordingly.Â 
I donâ€™t even really care what forms of re-use are counted, re-use in
research, re-use in education, in commerce, in industry, in policy
development. ORC is deliberately -- very deliberately -- intended to
hack the impact factor system by featuring highly re-usable tools that
will gain lots of citations. We need more of these hacks.

I think this shift is occurring. Itâ€™s not widely know just how close
UK science funding went to being slashed in the comprehensive spending
review. That it wasnâ€™t was due to a [highly coherent and well
organized campaign](http://scienceisvital.org.uk/) that convinced
ministers and treasury that the re-use of UK research outputs generated
enormous value, both economic, social and educational for the country
and indeed globally. That the [Sloan Digital Sky
Survey](http://www.sdss.org/) was available in a form that could be
re-used to support the development of something like [Galaxy
Zoo](http://www.galaxyzoo.org/) played a part in this. The headlong rush
of governments worldwide to release their data is a massive effort to
realize the potential value of the re-use of that data.

This change in focus is coming. It will no longer be enough in science
to just publish. As David Willetts said in \[answer to a question\] in
his first policy speech, â€œIâ€™m very much in favour of peer review,
but I worry when the only form of review is for journalsâ€?.Â 
Government wants evidence of wider use. They call it impact, but its
basically re-use. The policy changes are coming, the data sharing
policies, the public engagement policies, the impact assessments. Just
showing outputs will no be enough, showing that youâ€™ve configured
those outputs so that the potential for re-use is maximized will be an
assumption of receiving funding.

[William Gibson](http://en.wikipedia.org/wiki/William_Gibson) said the
future is already here, its just unevenly distributed. [They Might Be
Giants](http://en.wikipedia.org/wiki/They_Might_Be_Giants) asked, not
quite in response, â€œbut whereâ€™s my jetpack?â€? Â The jetpacks, the
tools, are around us and being developed if you know where to look. Best
practice is unevenly distributed both in science and in software
development but itâ€™s out there if you want to go looking. The
motivation to adopt it? The world around us is changing. The
expectations of the people who fund us are changing. Best practice in
code and in science have an awful lot in common. If you can master one
you will have to tools to help you with the other. And if you have both
then youâ€™ll be well positioned to ride the wave of change as it sweeps
by.

*  
*

::: {.zemanta-pixie}
[![Enhanced by
Zemanta](http://img.zemanta.com/zemified_e.png?x-id=3f3ef3d6-4603-482d-8c79-39ff935444fe){.zemanta-pixie-img
decoding="async"}](http://www.zemanta.com/ "Enhanced by Zemanta"){.zemanta-pixie-a}[]{.zem-script
.more-related .pretty-attribution .paragraph-reblog}
:::


<em>
 The following is the text from which I spoke today at the .Astronomy conference. I think there is some video available on the .Astronomy UStream account and I also have audio which I will put up somewhere soon.
</em>
Thereâ€™s a funny thing about the science and coding communities. Each seems to think that the other has all the answers.

Rogue Scholar Beiträge

Best practice in Science and Coding. Holding up a mirror.