Cameron Neylon

There has been a lot of recent discussion about the relative importance
of Open Source and Open Data
([Friendfeed](http://friendfeed.com/e/4a8c5012-ee18-2b72-390a-ca3298b3a57c/open-data-is-more-important-than-open/),
[Egon
Willighagen](http://chem-bla-ics.blogspot.com/2009/03/open-data-versus-capatalism.html),
[Ian
Davis](http://www.google.co.uk/url?sa=t&source=web&ct=res&cd=1&url=http%3A%2F%2Fiandavis.com%2Fblog%2F2009%2F03%2Fopen-data-open-source&ei=aPfUSd7GEaWsjAfA8ZnrDg&usg=AFQjCNF--83L-d02v50DIzj8h6mV4q__pg&sig2=jIUzMqWW5glEKHwbL4148w)).
I don't fancy recapitulating the whole argument but following a
discussion on Twitter with [Glyn Moody](http://twitter.com/glynmoody)
this morning \[[1](http://twitter.com/glynmoody/statuses/1437524981),
[2](http://twitter.com/CameronNeylon/status/1437541770),
[3](http://twitter.com/glynmoody/statuses/1437548842),
[4](http://twitter.com/CameronNeylon/status/1437558740),
[5](http://twitter.com/CameronNeylon/status/1437563815),
[6](http://twitter.com/glynmoody/statuses/1437575352),
[7](http://twitter.com/glynmoody/statuses/1437662765),
[8](http://twitter.com/CameronNeylon/status/1437727643)\] I think there
is a way of looking at this with a slightly different perspective. But
first a short digression.

I attended a [workshop late last year on Open
Science](http://www.okfn.org/wiki/OpenScience/Workshop) run by the [Open
Knowledge Foundation](http://www.okfn.org). I spent a significant part
of the time arguing with Rufus Pollock about data licences, an argument
that is still going on. One of Rufus' challenges to me was to commit to
working towards using only Open Source software. His argument was that
there wasn't really any excuses any more. [Open
Office](http://www.openoffice.org/) could do the job of [MS
Office](http://office.microsoft.com/en-us/default.aspx),
[Python](http://python.org) with [SciPy](http://scipy.org) was up to the
same level as [MatLab](http://en.wikipedia.org/wiki/MATLAB), and
anything specialist needed to be written anyway so should be open source
from the off.

I took this to heart and I have tried, I really have tried. I needed a
new computer and, although I got a Mac (not really ready for Linux yet),
I loaded it up with Open Office, I haven't yet put my favourite data
analysis package on the computer
([Igor](http://www.wavemetrics.com/products/IGORPro/igorpro.htm) if you
must know), and have been working in Python to [try to get some stuff up
to speed](http://github.com/cameronneylon/sas/tree/master). But I have
to ask whether this is the best use of my time. As is often the case
with my arguments this is a return on investment question. I am paid by
the taxpayer to do a job. At what point is the extra effort I am putting
into learning to use, or in some cases fight with, new tools cost more
than the benefit that is gained, by making my outputs freely available?

Sometimes the problems are imposed from outside. I spent a good part of
yesterday battling with an appalling, password protected,
macroed-to-the-eyeballs Excel document that was the required format for
me to fill in a form for an application. The file crashed Open Office
and only barely functioned in Mac Excel at all. Yet it was required, in
that format, before I could complete the application. Sometimes the
software is just not up to scratch. Open Office Writer is fine, but the
presentation and spreadsheet modules are, to be honest, a bit ropey
compared to the commercial competitors. And with a Mac I now have
Keynote which is just so vastly superior that I have now transferred
wholesale to that. And sometimes it is just a question of time. Is it
really worth me learning Python to do data analysis that I could knock
in Igor in a tenth of the time?

In this case the answer is, probably yes. Because it means I can do more
with it. There is the potential to build something that logs process the
way I want to , the potential to convert it to run as a web service. I
could do these things with other OSS projects as well in a way that I
can't with a closed product. And even better because there is a big open
community I can ask for help when I run into
[problems](http://statingthebleedingobvious.wordpress.com/2009/03/16/problems-with-creating-a-new-matplotlib-scalebase-class/).

It is easy to lose sight of the fact that for most researchers software
is a means to an end. For the Open Researcher what is important is the
ability to reproduce results, to criticize and to examine. Ideally this
would include every step of the process, including the software. But for
most issues you don't need, or even want, to be replicating the work
right down to the metal. You wouldn't after all expect a researcher to
be forced to run their software on an open source computer, with an open
source chipset. You aren't necessarily worried what operating system
they are running. What you are worried about is whether it is possible
read their data files and reproduce their analysis. If I take this just
one step further, it doesn't matter if the analysis is done in MatLab or
Excel, as long as the files are readable in Open Office and the analysis
is described in sufficient detail that it can be reproduced or
re-implemented.

Lets be clear about this: it would be better if the analysis were done
in an OSS environment. If you have the option to work in an OSS
environment you can also save yourself time and effort in describing the
process and others have a much better chances of identifying the sources
of problems. It is not good enough to just generate an Excel file, you
have to generate an Excel file that is readable by other software (and
here I am looking at the increasing number of instrument manufacturers
providing software that generates so called Excel files that often
aren't even readable in Excel). In many cases it might be easier to work
with OSS so as to make it easier to generate an appropriate file. But
there is another important point; if OSS generates a file type that is
undocumented or worse, obfuscated, then that is also unacceptable.

Open Data is crucial to Open Research. If we don't have the data we have
nothing to discuss. Open Process is crucial to Open Research. If we
don't understand how something has been produced, or we can't reproduce
it, then it is worthless. Open Source is not necessary, but, if it is
done properly, it can come close to being sufficient to satisfy the
other two requirements. However it can't do that without Open Standards
supporting it for documenting both file types and the software that uses
them.

The point that came out of the conversation with Glyn Moody for me was
that it may be more productive to focus on our ability to re-implement
rather than to simply replicate. Re-implementability, while an awful
word, is closer to what we mean by replication in the experimental world
anyway. Open Source is probably the best way to do this in the long
term, and in a perfect world the software and support would be there to
make this possible, but until we get there, for many researchers, it is
a better use of their time, and the taxpayer's money that pays for that
time, to do that line fitting in Excel. And the damage is minimal as
long as source data and parameters for the fit are made public. If we
push forward on all three fronts, Open Data, Open Process, and Open
Source then I think we will get there eventually because it is a more
effective way of doing research, but in the meantime, sometimes, in the
bigger picture, I think a shortcut should be acceptable.


There has been a lot of recent discussion about the relative importance of Open Source and Open Data (Friendfeed, Egon Willighagen, Ian Davis). I don’t fancy recapitulating the whole argument but following a discussion on Twitter with Glyn Moody this morning [1, 2, 3, 4, 5, 6, 7, 8] I think there is a way of looking at this with a slightly different perspective. But first a short digression.

Open Data, Open Source, Open Process: Open Research

On Monday 1 September we had a one day workshop in Southampton
discussing the issues that surround 'Open Science'. This was very free
form and informal and I had the explicit aim of getting a range of
people with different perspectives into the room to discuss a wide range
of issues, including tool development, the social and career structure
issues, as well as ideas about standards and finally, what concrete
actions could actually be taken. You can find live blogging and other
commentary in the associated [Friendfeed
room](http://friendfeed.com/rooms/soton-open-science-workshop-1-9-08)
and information on who attended as well as links to many of the
presentations on the [conference
wiki](http://sotonopensciworkshop.wikispaces.com/).

Broadly speaking the day was divided into three chunks, the first was
focussed on tools and services and included presentations on
[MyExperiment](http://myexperiment.org),
[Mendeley](http://www.mendeley.com/),
[Chemtools](http://chemtools.chem.soton.ac.uk/), and [Inkspot
Science](http://www.inkspotscience.com/). Branwen Hide of Research
Information Network has written [more on this
part](http://www.rin.ac.uk/node/440). Given that the room contained more
than the usual suspects the conversation focussed on usability and
interfaces rather than technical aspects although there was a fair bit
of that as well.

The second portion of the day revolved more around social challenges and
issues. [Richard Grant](http://network.nature.com/people/rpg/blog/)
presented his experience of blogging on an official university
sanctioned site and the value of that for both outreach and education.
One point he made was that the 'lack of adoption problem' seen in
science just doesn't seem to exist in the humanities. Perhaps this is
because scientists don't generally see 'writing' as a valuable thing in
its own right. Certainly there is a preponderance of scientists who
happen also to see themselves as writers on [Nature
Network](http://network.nature.com).

[Jennifer Rohn](http://network.nature.com/people/UE19877E8/blog)
followed on from Richard, and objected to my characterising her
presentation as "the skeptic's view". A more accurate characterisation
would have been "I'd love to be open but at the moment I can't: This is
what has to change to make it work". She presented a great summary of
the proble, particularly from the biological scientist's point of view
as well as potential solutions. Essentially the problem is that of the
'Minimum Publishable Unit' or research quantum as well as what 'counts'
as publication. Her main point was that for people to be prepared to
publish material that falls short of a full paper they need to get some
proportional credit for that. This folds closely into the discussion of
what can be cited and what should be cited in particular contexts. I
have used the phrase 'data sized peg into a paper shaped hole' to
describe this in the past.

After lunch [Liz Lyon](http://www.ukoln.ac.uk/ukoln/staff/e.j.lyon/)
from [UKOLN](http://www.ukoln.ac.uk/) talked about curation and long
term archival storage which lead into an interesting discussion about
the archiving of blogs and other material. Is it worth keeping? One
answer to this was to look at the real interest today in diaries from
the second world war and earlier from 'normal people'. You don't
necessarily need to be a great scientist, or even a great blogger, for
the material to be of potential interest to historians in 50-100 years
time. But doing this properly is hard -- in the same way that
maintaining and indexing data is hard. Disparate sites, file formats,
places of storage, and in the end whose blog is it actually?
Particularly if you are blogging for, or recording work done at, a
research institution.

The final session was about standards or 'brands'. Yaroslav Nikolaev
talked about semantic representations of experiments. While important it
was probably a shame in the end we did this at the end of the day
because it would have been helpful to get more of the non-techie people
into that discussion to iron out both the communication issues around
semantic web as well as describing the real potential benefits. This
remains a serious gap -- the experimental scientists who could really
use semantic tools don't really get the point, and the people developing
the tools don't communicate well what the benefits are, or in some cases
(not all I hasten to add!) actually build the tools the experimentalists
want.

I talked about the possibility of a 'certificate' or standard for Open
Science, and the idea of an organisation to police this. It would be
safe to say that, while people agreed that clear definitions would be
hepful, the enhusiasm level for a standards organisation was pretty much
zero. There are more fundamental issues of actually building up enough
examples of good practice, and working towards identifying best practice
in open science, that need to be dealt with before we can really talk
about standards.

On the other hand the idea of 'the fully supported' paper got immediate
and enthusiastic support. The idea here is deceptively simple, and has
been discussed elsewhere; simply that all the relevant supporting
information for a paper (data, detailed methodology, software tools,
parameters, database versions etc. as well as access to required
materials at reasonable cost) should be available for any published
paper. The challenge here lies in actually recording experiments in such
a way that this information can be provided. But if all of the record is
available in this form then it can be made available whenever the
researcher chooses. Thus by providing the tools that enable the fully
supported paper you are also providing tools that enable open science.

Finally we discussed what we could actually do: [Jean-Claude
Bradley](http://usefulchem.blogspot.com) discussed the idea of an [Open
Notebook Science challenge](http://onschallenge.wikispaces.com/) to
raise the profile of ONS (this is now setup -- more on this to follow).
Essentially a competition type approach where individuals or groups can
contribute to a larger scientific problem by collecting data -- where
the teams get judged on how well they describe what they have done and
how quickly they make it available.

The most specific action proposed was to draft a 'Letter to Nature'
proposing the idea of the fully supported paper as a submission
standard. The idea would be to get a large number of high profile
signatories on a document which describesÂ  a concrete step by step plan
to work towards the final goal, and to send that as correspondence to a
high profile journal. I have been having some discussions about how to
frame such a document and hope to be getting a draft up for discussion
reasonably soon.

Overall there was much enthusiasm for things Open and a sense that many
elements of the puzzle are falling into place. What is missing is
effective coordinated action, communication across the whole community
of interested and sympathetic scientsts, and critically the high profile
success stories that will start to shift opinion. These ought to, in my
opinion, be the targets for the next 6-12 months.


On Monday 1 September we had a one day workshop in Southampton discussing the issues that surround ‘Open Science’. This was very free form and informal and I had the explicit aim of getting a range of people with different perspectives into the room to discuss a wide range of issues, including tool development, the social and career structure issues, as well as ideas about standards and finally, what concrete actions could actually be taken.

The Southampton Open Science Workshop – a brief report

*For the [Open Science workshop](http://psb09openscience.wordpress.com/)
at the [Pacific Symposium on Biocomputing](http://psb.stanford.edu/) I
wrote a very long essay as an introductory paper. It turned out that
this was far too long for the space available so an extremely shortened
version was submitted for the symposium proceedings. I thought I would
post the full length essay in installments here as a prelude to cleaning
it up and submitting to an appropriate journal.  
*

**Introduction**

Openness is arguably the great strength of the scientific method. At its
core is the principleÂ that claims and the data that support them are
placed before the community for examination and critique. Through open
examination and critical analysis models can be refined, improved, or
rejected. Conflicting data can be compared and the underlying
experiments and methodology investigated to identify which, if any, is
more reliable. While individuals may not always adhere to the highest
standards, the community mechanisms of review, criticism, and
integration have proved effective in developing coherent andÂ useful
models of the physical world around us. As Lee Smolin of the Perimeter
Institute for Theoretical Physics recently put it, "*we argue in good
faith from shared evidence to shared
conclusions*"\[[1](http://pirsa.org/08090035/)\]. It is an open approach
that drives science towards an understanding which, while never perfect,
nevertheless enables the development of sophisticated technologies with
practical applications.

The Internet and the World Wide Web provide the technical ability to
share a much wider range of both the evidence and the argument and
conclusions that drive modern research. Data, methodology, and
interpretation can also be made available online at lower costs and with
lower barriers to access than has traditionally been the case. Along
with the ability to share and distribute traditional scientific
literature, these new technologies also offer the potential for new
approaches. Wikis and blogs enable geographically and temporally
widespread collaborations, the traditional journal club can now span
continents with online book marking tools such as Connotea and
CiteULike, and the smallest details of what is happening in a laboratory
(or on Mars \[[2](http://twitter.com/MarsPhoenix)\]) can be shared via
instant messaging applications such as Twitter.

The potential of online tools to revolutionise scientific communication
and their ability to open up the details of the scientific enterprise so
that a wider range of people can participate is clear. In practice,
however, the reality has fallen far behind the potential. This is in
part due to a need for tools that are specifically designed with
scientific workflows in mind, partly due to the inertia of
infrastructure providers with pre-Internet business models such as the
traditional "subscriber pays" print literature and, to some extent,
research funders. However it is predominantly due to cultural and social
barriers within the scientific community. The prevailing culture of
academic scientific research is one of possession -- where control over
data, methodological secrets, and exploitation of results are paramount.
The tradition of Mertonian Science has receded, in some cases, so far
that principled attempts to reframe an ethical view of modern science
can seem charmingly naive.

It is in the context of these challenges that the movement advocating
more openness in science must be seen. There will always be places where
complete openness is not appropriate, such as where personal patient
records may be identifiable, where research is likely to lead to
patentable (and patent-worthy) results, or where the safety or privacy
of environments, study subjects, or researchers might be compromised.
These, however are special instances for which exceptional cases can be
made, and not the general case across the whole of global research
effort. Significant steps forward such as funder and institutional
pre-print deposition mandates and the adoption of data sharing policies
by UK Research Councils must be balanced against the legal and
legislative attempts to overturn the NIH mandate and widespread
confusion over what standards of data sharing are actually required and
how they will be judged and enforced. Nonetheless there is a growing
community interested in adopting more open practices in their research,
and increasingly this community is developing as a strong voice in
discussions of science policy, funding, and publication.Â  The aim of
this workshop is to strengthen this voice by focusing the attention of
the community on areas requiring technical development, the development
and implementation of standards, both technical and social, and
identification and celebration of success.

**Why we need open science -- Open Access publication, Open Data, and
Open Process**

The case for taxpayer access to the taxpayer funded peer reviewed
literature was made personally and directly in Jonathon Eisen's first
editorial for PLoS Biology
\[[3](http://dx.doi.org/10.1371/journal.pbio.0060048)\].

\[...describing the submission of a paper to PLoS Biology as an
'experiment'...\] But then, while finalizing the paper, a two-month-long
medical nightmare ensued that eventually ended in the stillbirth of my
first child. While my wife and I struggled with medical mistakes and
negligence, we felt the need to take charge and figure out for ourselves
what the right medical care should be. And this is when I experienced
the horror of closed-access publishing. For unlike my colleagues at
major research universities that have subscriptions to all journals, I
worked at a 300-person nonprofit research institute with a small
library. So there I wasâ€"a scientist and a taxpayerâ€"desperate to read
the results of work that I helped pay for and work that might give me
more knowledge than possessed by our doctors. And yet either I could not
get the papers or I had to pay to read them without knowing if they
would be helpful. After we lost our son, I vowed to never publish in
non-OA journals if I was in control. \[...\]

Eisen JA (2008) PLoS Biology 2.0. PLoS Biol 6(2): e48
[doi:10.1371/journal.pbio.0060048](http://dx.doi.org/10.1371/journal.pbio.0060048)

As a scientist in a small institution he was unable to access the
general medical literature. More generally, as a US taxpayer he was
unable to access the outputs of US government funded research or indeed
of research funded by the governments of other countries. The general
case for enabling access of both the general public, scientists in less
well funded institutions, and in the developing world has been accepted
by most in principle. While there are continuing actions being taken to
limit the action of the NIH mandate by US publishers a wide range of
research institutions have adopted deposition mandates. There remains
much discussion about routes to open access with the debate over 'Green'
and 'Gold' routes continuing as well as an energetic ongoing debate
about the stability and viability of the business models of various open
access journals. However it seems unlikely that the gradual increase in
number and impact of open access journals is likely to slow or stop
soon. The principle that the scientific literature *should* be available
to all has been won. The question of *how* best to achieve that remains
a matter of debate.

A similar case to that for access to the published literature can also
be made for research data. At the extremes, withholding data could lead
toÂ preventable deaths or severely reduced quality of life for patients.
Andrew Vickers, in a hard hitting New York Times essay
\[[4](http://blog.openwetware.org/scienceintheopen/wp-admin/post-new.php#%20http://www.nytimes.com/2008/01/22/health/views/22essa.html?_r=1%20#)\]
dissected the reasons that medical scientists give for not making data
from clinical cancer trials available; data that could, in aggregate,
provide valuable insights into enhancing patient survival time and
quality of life. He quotes work by John Kirwan (Bristol University)
showing that three quarters of researchers in one survey opposed sharing
data from clinical trials. While there may be specific reasons for
retaining specific types of data from clinical trials, particularly in
small specialised cases where maintaining the privacy of participants is
difficult or impossible, it seems unarguable that the interests of
patients and the public demand that such data be available for re-use
and analysis. This is particularly the case where the taxpayer has
funded these trials, but for other funders, including industrial
funders, there is a public interest argument for making clinical trial
data public in particular.

In other fields the case for data sharing may seem less clear cut. There
is little obvious damage done to the general public by not making the
details of research available. However, while the argument is more
subtle, it is similar to that for clinical data. There the argument is
that reanalysis and aggregation can lead to new insights with an impact
on patient care. In non-clinical sciences this aggregation and
re-analysis leads to new insights, more effective analysis, and indeed
new types of analysis. The massive expansion in the scale and ambition
of biological sciences over the past twenty years is largely due to the
availability of biological sequence, structural, and functional data in
international and freely available archives. Indeed the entire field of
bioinformatics is predicated on the availability of this data. There is
a strong argument to be made that the failure of the chemical sciences
to achieve a similar revolution is due to the lack of such publicly
available data. Bioinformatics is a highly active and widely practiced
field of science. By comparison, chemoinformatics is marginalised, and,
what is most galling to those who care for the future of chemistry,
primarily driven by the needs and desires of biological scientists.
Chemists for the most part haven't grasped the need because the
availability of data is not part of their culture.

High energy particle physics by contrast is necessarily based on a
community effort; without strong collaboration, communication, and
formalised sharing of the details of what work is going on the research
simply would not happen. Astronomy, genome sequencing, and protein
crystallography are other fields where there is a strong history, and in
some cases formalized standards of data sharing. While there are
anecdotal cases of 'cheating' or bending the rules, usually to prevent
or restrict the re-use of data, the overall impact of data sharing in
these areas is generally seen as positive, leading to better science,
higher data quality standards, and higher standards of data description.
Again, to paraphrase Smolin, where the discussion proceeds from a shared
set of evidence we are more likely to reach a valid conclusion. This is
simply about doing better science by improving the evidence base.

The final piece of the puzzle, and in many ways the most socially and
technically challenging is the sharing of research procedures. Data has
no value in and of itself unless the process used to generate it is
appropriate and reliable. Disputes over the validity of claims are
rarely based on the data themselves but on the procedures used either to
collect them or those used to process and analyse them. A widely
reported recent case turned on the details of how a protein was
purified; whether with a step or gradual gradient elution. This detail
of procedure led laboratories to differing results, a year of wasted
time for one researcher, and ultimately the retraction of several high
profile papers \[refs -- nature feature, retractions, original paper
etc\]. Experimental scientists generally imagine that in the
computational sciences where a much higher level of reproducibility and
the ready availability of code and subversion repositories makes sharing
and documenting material relatively straightforward, would have much
higher standards. However, a recent paper \[6\] by Ted Pedersen
(University of Minnesota, Duluth) -- with the wonderful title
'Empiricism is not a matter of faith' -- criticized the standards of
both code documentation and availability. He makes the case that working
with the assumption that you will make the tools available to others not
only allows you to develop better tools, and makes you popular in the
community, but also improves the quality of your own work.

And this really is the crux of the matter. If the central principle of
the scientific method is open analysis and criticism of claims then
making the data and process and conclusions avalable and accessible is
just doing good science. While we may argue about the timing of release
or the details of 'how raw' available data needs to be or the file
formats or ontologies used to describe it there can be no argument that
if the scientific record is to have value it must rest on an accessible
body of relevant evidence. Scientists were doing mashups long before the
term was invented; mixing data from more than one source; reprocessing
it to provide a different view. The potential of online tools to help to
do this better is massive, but the utility of these tools depends on the
sharing of data, workflows, ideas, and opinions.

There are broadly three areas for development that are required to
enable the more widespread adoption of open practice by research
scientists. The first is the development of tools that are designed for
scientists. While many of the general purpose tools and services have
been adopted by researchers there are many cases where specialised
design or adaptation is required for the specific needs of a research
environment. In some cases the needs of research willpush development in
specific areas, such as controlled vocabularies, beyond what is being
done in the mainstream. The second, and most important area involves the
social and cultural barriers within various research communities.These
vary widely in type and importance across different fields and
understanding and overcoming the fears as well as challenging entrenched
interests will be an important part of the open science programme.
Finally, there is a value and a need to provide top-down guidance in the
form of policies and standards. The vagueness of the term 'Open Science'
means that while it is a good banner there is a potential for confusion.
Standards, policies, and brands can provide clarity for researchers, a
clear articulation of aspirations (and a guide to the technical steps
required to achieve them), and the support required to help people
actually make this happen in their own research.

*[Part
II](http://cameronneylon.net/blog/a-personal-view-of-open-science-part-ii-tools/)
will cover the issues around tools for Open Science*

**References**

1.  Smolin L (2008), Science as an ethical community, PIRSA ID#08090035,
    <http://pirsa.org/08090035/>
2.  Mars Phoenix on Twitter, <http://twitter.com/MarsPhoenix>
3.  Eisen JA (2008) PLoS Biology 2.0. PLoS Biol 6(2): e48
    [doi:10.1371/journal.pbio.0060048](http://dx.doi.org/10.1371/journal.pbio.0060048)
4.  Vickers A (2008),
    <http://www.nytimes.com/2008/01/22/health/views/22essa.html?_r=1>
5.  Pedersen T (2008), Computational Linguistics, Volume 34, Number 3,
    pp. 465-470, [Self
    archived](http://www.d.umn.edu/%7Etpederse/Pubs/pedersen-last-word-2008.pdf).


*For the Open Science workshop at the Pacific Symposium on Biocomputing I wrote a very long essay as an introductory paper. It turned out that this was far too long for the space available so an extremely shortened version was submitted for the symposium proceedings. I thought I would post the full length essay in installments here as a prelude to cleaning it up and submitting to an appropriate journal.

A personal view of Open Science – Part I

I hold no particular candle for traditional peer review. I think it is
inefficient, poorly selective, self reinforcing, often poorly done, and
above all, far too slow. However I also agree that it is the least worst
system we have available to us.Â  Thus far, no other approaches have
worked terribly well, at least in the communication of science research.
And as the incumbent for the past fifty years or so in the post of
'generic filter' it is owed some respect for seniority alone.

So I am considering writing a fellowship proposal that would be based
around the idea of delivering on the Open Science Agenda via three
independent projects, one focussed on policy and standards development,
one on delivering a shareable data analysis pipeline for small angle
scattering as an exemplar of how a data analysis *process* can be
shared, and a third project based around building the infrastructure for
embedding real science projects involving chemistry and drug discovery
in educational and public outreach settings. I think I can write a
pretty compelling case around these three themes and I think I would be
well placed to deliver on them, particularly given the collaborative
support networks we are already building in these areas.

The thing is I have no conventional track record in these areas. There
are a bunch of papers currently being written but none that will be out
in print by the time the application is supposed to go in. My recorded
contribution in this area is in blog posts, blog comments, presentations
and other material, all of which are available online. But none of which
are peer-reviewed in the conventional sense.

One possibility is to make a virtue of this -- stating that this is a
rapidly moving field -- that while papers are in hand and starting to
come out that the natural medium for communication with the specific
community is online through blogs and other media. There is an argument
that conventional peer review simply does not map on to the web of data,
tools, and knowledge that is starting to come together and that
measuring a contribution in this area by conventional means is simply
misguided.Â  All of which I agree with in many ways.

I just don't think the referees will buy it.

Which got me thinking. It's not just me, many of the seminal works for
the Open Science community are not peer reviewed papers. [Bill
Hooker](http://www.sennoma.net)'s three parter
\[[1](http://3quarksdaily.blogs.com/3quarksdaily/2006/10/the_future_of_s_1.html),
[2](http://3quarksdaily.blogs.com/3quarksdaily/2006/11/the_future_of_s.html),
[3](http://3quarksdaily.blogs.com/3quarksdaily/2007/01/the_future_of_s.html)\]
at Three Quarks Daily comes to mind, as does Jean-Claude's
[presentation](http://precedings.nature.com/documents/39/version/1) on
Nature Precedings on Open Notebook Science, Michael Nielsen's essay
*[The Future of Science](http://michaelnielsen.org/blog/?p=448)*, and
Shirley Wu's [*Envisioning the scientific community as One Big
Lab*](http://onebiglab.blogspot.com/2008/04/envisioning-scientific-community-as-one.html)
(along with many others). It seems to me that these *ought* to have the
status of peer reviewed papers which raises the question. We are a
community of peers, we can referee, we can adopt some sort of standard
of signficance and decide to apply that selectively to specific works
online. So why can't we *make* them peer reviewed?

What would be required? Well a stable citation obviously, so probably a
DOI and some reasonably strong archival approach, probably using
[WebCite](http://www.webcitation.org/).Â  There would need to be a clear
process of peer review, which need not be anonymous, but there would
have to be a clear probity trail to show that an independent editor or
group of referees made a decision and that appropriate revisions had
been made and accepted. The bar for acceptance would also need to be set
pretty high to avoid the charge of simply rubber stamping a bunch of
online material. I don't think open peer review is a problem for this
community so many of the probity questions can be handled by simply
having the whole process out in the open.

One model would be for an item to be submitted by posting a link on a
new page on an independent Wiki . This would then be open to peer
review. Once three (five?) independent reviewers had left comments and
suggestions -- and a version of the document created that satisfied them
posted -- then the new version could be re-posted at the author's site,
in a specified format which would include the DOI and arhival links,
along with a badge that would be automatically aggregated to create the
index *a la* [researchblogging.org](http://researchblogging.org). There
would need to be a charge, either for submission or acceptance --
submission would keep volume down and (hopefully) quality up.

How does this differ from setting up a journal? Well two major things --
one is that the author remains the publisher so the costs of publication
*per se* are taken out of the equation. This is important as it keeps
costs down -- not zero, there is still the cost of the DOI and (even if
it is donated) the time of editors and referees in managing the process
and giving a stamp of authority. The main cost is in maintaining some
sort of central index and server pointing out at the approved items. It
would also be appropriate to support WebCite if that is the backstop
archive. But the big costs for journals are in providing storage that is
stable in the long term and managing peer review. If the costs of
storage are offloaded *andÂ * the peer review process can be self
organised then the costs drop significantly.

The second major advantage is that, as a community we already do a lot
of this, looking over blog posts, linking to presentations, writing
commentary or promoting them on FriendFeed. The reason why
[ArXiv](http://arxiv.org) worked was that there was already a culture of
preprints amongst that community. The reason why commenting, rating,Â 
and open peer review trials have not been as successful as people had
hoped is because there is no pre-existing culture of doing these things.
We already have a culture of open peer review in our community. Is it
worth formalising it for the really high quality material that's already
out there?

I am aware that this goes against many of the principles of open and
continuous review that many of you hold dear but I think it could serve
two useful purposes. First it means that members of the community,
particularly younger members, can bolster their CV with peer reviewed
papers. Come the revolution this won't matter but we're not there yet.
Making these contributions tangible for people could be quite powerful.
Secondly it takes the important material out of the constant stream of
objects flitting past on our screens and gives them a static (I won't
say permanent) priviledged place as part of the record of this field.Â 
Many of them perhaps already have this but I think there is a value in
formalising it. Is it worth considering? This proposal is out for
review.

 


I hold no particular candle for traditional peer review. I think it is inefficient, poorly selective, self reinforcing, often poorly done, and above all, far too slow. However I also agree that it is the least worst system we have available to us.Â  Thus far, no other approaches have worked terribly well, at least in the communication of science research.

Messaggi di Rogue Scholar

Open Data, Open Source, Open Process: Open Research

The Southampton Open Science Workshop – a brief report

A personal view of Open Science – Part I

Thinking about peer review of online material: The Peer Reviewed Journal of Open Science Online