University of Glasgow

Roderic Page

This post is likely to seem somewhat off the wall, given the rush to
getting everything in the cloud, but it\'s Friday, so let\'s give it a
whirl.  
  
One idea I\'ve been toying with is dispensing with relational databases,
wikis, etc. and just storing taxonomic data using files and folders on a
disk. There are several reasons for this:

- File system naturally enforces hierachy
- There are existing systems for putting files and folders under version
  control (e.g., CVS, Subversion, git)
- Native text and image editors handily beat web-based ones
- Some file systems have great tools for searching on metadata (e.g.,
  \"smart folders\" and Spotlight on Mac OS X)
- Some of the visualisations that we would like for classifications
  (such as [treemaps](http://grandperspectiv.sourceforge.net/)) already
  exist in very polished form for viewing file systems

By way of background, I\'ve been prompted to think along these lines by
[David Shorthouse\'s
observation](http://ispiders.blogspot.com/2007/10/taxonomic-consensus-as-software.html)
that we could place a taxonomic hierachy under version control (e.g.,
[github](http://github.com)) and deal with changes/multiple versions
that way. I\'ve also been inspired by tools such as
[couchdb](http://couchdb.apache.org/%22%22), which is a schema-less
database that one can talk to directly via HTTP. This reflects a trend
where people are starting to exploit the untapped power in some basic,
well-known technologies (such as the HTTP protocol), avoiding the need
for lots of middle-ware in the process (why write stuff in
Java/Ruby/PHP, etc. when HTTP GET, PUT, POST, etc. cover the bases?).
Another inspiration is [Dropbox](http://getdropbox.com), which enables
replication of files across multiple machines and the web. The web
interface to Dropbox is very clean, and essentially mirrors the local
folder structure.  
  
So, in some ways this probably sounds silly, and closely resembles the
naive way many of us started making digital versions of taxonomies, and
it will have many database people rolling their eyes and muttering about
\"data consistency\" and \"queries\". But, a key thing to remember is
that the file system is a database that resides under a graphical user
interface, and it maintains some forms of consistency that classical
relational databases are poor at handling. For example, file systems
enforce hierarchical consistency (if I move a folder to another folder,
all the files and folders below that folder move as well). Of course, we
can program this with a relational database, but our track record in
doing this is pretty miserable. I\'ve found inconsistencies in versions
of ITIS (haven\'t checked recently), and last years\' Catalogue of Life
database had all sorts of orphans lurking in the tree table.  
  
Then there\'s the GUI. If I write a taxonomic database in the classical
way, I need to write code to talk to the database, edit records, support
user authentication, data versioning, etc. If I use the file system, I
get this pretty much for free. Authentication? It\'s called the login
screen. Versioning? I put it in a public repository like Google Code or
github, and that takes care of that (plus I get online authentication
for free). Editing? Well, I can drag and drop items onto folders, and I
can open them in native editors.  
  
What I envisage is replicating a taxonomic hierarchy on disk, and
representing key-value pairs of attributes (such as taxon name
authorship, bibliographic details) as text files where the name of the
file is the key (e.g., **publishedIn**) and the content of the file is
the value (e.g., **doi:10.1590/S0101-81752005000300004**). I could add
images and PDFs, and the neat thing is that they have lots of useful
metadata embedded inside (where, arguably, it belongs).  
  
I\'m also toying with the idea of using symbolic links (Windows users,
look away now) to represent relationships such as basionym links to
original names.  
  
This is all a bit half-baked at present, but it seems worth pursing. One
could argue that having a full taxonomic hierachy is overkill (and
raises the issue of which one to use), but binomial names are themselves
hierachical (species epithet nested inside genus name), so we need some
degree of hierarchy anyway. I like the idea that copying a folder called
\"behreae\" in the folder \"Pinnixa\" and placing the copy under
\"Austinixa\", then within Austinixa/behreae adding a symbolic link to
Pinnixa/behreae pretty much takes care of synonomy. I also like the idea
that one could download an entire taxonomy, and using just the native
tools on your computer, edit and annotate it, then merge changes with a
remote copy. It makes mamnaging the data little different from writing
code.  
  
In practise we\'ll want to add some things. It would be nice to have a
web interface for browsing, but this could be as trivial as having a
script that read the contents of a folder, display folders as HTML
links, and list the files (keys) and their contents (values) in the web
page.  
  
Perhaps this is a little silly, but I like the idea of having data on my
machine that is trivially easy to edit. I also like the idea of getting
functionality for free, rather than having to invent it from scratch.


This post is likely to seem somewhat off the wall, given the rush to getting everything in the cloud, but it's Friday, so let's give it a whirl. One idea I've been toying with is dispensing with relational databases, wikis, etc. and just storing taxonomic data using files and folders on a disk.

Taxonomy on a hard disk

Cameron Neylon

[![Sign at The Googleplex. Google limited access to Xenu.net, in March
2002.](http://upload.wikimedia.org/wikipedia/en/thumb/e/ee/Googleplex_Welcome_Sign.jpg/202px-Googleplex_Welcome_Sign.jpg){decoding="async"}](http://en.wikipedia.org/wiki/Image:Googleplex_Welcome_Sign.jpg)I
realised the other day that I havenâ€™t written an exciteable blog post
about getting an invitation to
[SciFoo](http://en.wikipedia.org/wiki/Science_Foo_Camp "Science Foo Camp"){rel="wikipedia"}!
The reason for this is that [I got overexcited over on
FriendFeed](http://friendfeed.com/e/0b4e74bd-9513-4c20-7330-3d51868290fb)
instead and havenâ€™t really had time to get my head together to write
something here. But in this post I want to propose a session and think
through what the focus and aspects of that might be.

I am a passionate advocate of two things that I think are intimately
related. I believe strongly in the need and benefits that will arise
from building, using, and enabling the effective search and processing
of a scientific data commons. I
\[[1](http://blog.openwetware.org/scienceintheopen/2008/04/16/the-science-exchange/),[2](http://blog.openwetware.org/scienceintheopen/2008/05/11/more-on-the-science-exchance-or-building-and-capitalising-a-data-commons/)\]
and others (including
[John](http://network.nature.com/blogs/user/wilbanks/2008/05/10/on-the-erosion-of-the-public-domain)
[Wilbanks](http://network.nature.com/blogs/user/wilbanks/2008/05/11/on-a-data-commons),
[Deepak Singh](http://feeds.feedburner.com/~r/mndoci/~3/288151920/), and
[Plausible
Accuracy](http://www.plausibleaccuracy.com/2008/05/12/data-should-be-public-domain-and-more-esoteric-blog-based-rasslin/))
have written on this quite a lot recently. The second aspect is that I
believe strongly in the need for effective useable and generic tools to
record science as it happens and to process that record so that others
can use it effectively. To me these two things are intimately related.
By providing the tools that enable the record to be created and
integrating them with the systems that will store and process the data
commons we can enable scientists to record their work better,
communicate it better, and make it available *as a matter of course* to
other scientists (not necessarily immediately I should add, but when
they are comfortable with it).

At last yearâ€™s SciFoo Chris DiBona ran a session called â€˜Give us the
Dataâ€™ \[Ed. As noted below Jon Trowbridge led the session with Chris
DiBona acting as 'session chair'\] Now I wasnâ€™t there but Deepak Singh
[wrote the session up in a blog
post](http://mndoci.com/blog/2007/08/09/scifoo-google-and-large-scientific-datasets/)
which is my main inspiration. The idea here seemed to be to send in big
data sets and that
[Google](http://en.wikipedia.org/wiki/Google "Google"){rel="wikipedia"}
could slurp these up for processing. Now the world has moved on from
last year and I believe we can propose something much more ambitious
*with the tools we have available*. We can build the systems, or at
least the prototypes of systems, that will be so useful for recording
and sharing data that people will want to use them. Google AppEngine,
Amazon Web Services, Freebase,
[CouchDB](http://www.couchdb.com/ "CouchDB"){rel="homepage"}, Twine, and
the growing efforts in actually making controlled vocabularies useful
and useable, as well as a growing web of repositories and services make
this look like an achievable target in a way that it wasnâ€™t twelve
months ago. Once we start building these repositories of data then there
will be a huge incentive both for scientists and big players such as
Google, Microsoft, Yahoo, IBM, and Amazon to look at how to effectively
process, re-process, and present this data. That will be exciting, but
first we need to build the tools and systems that will enable us to put
the data there.

What does this â€˜systemâ€™ look like. Iâ€™m going to suggest a possible
model, based on my non-technical understanding of how these things work.
I will probably get the details wrong (and please comment where I do).
Letâ€™s start at the storage end. We are only interested in open
repositories, where it is explicitly stated that data is in the [public
domain](http://en.wikipedia.org/wiki/Public_domain "Public domain"){rel="wikipedia"}
and therefore freely re-useable. Some of these repositories exist for
specific domains.
[ChemSpider](http://www.chemspider.com/ "ChemSpider"){rel="homepage"} ,
NMRShiftDB and crystallography databases in the chemistry domain,
GenBank,
[PDB](http://en.wikipedia.org/wiki/Protein_Data_Bank "Protein Data Bank"){rel="wikipedia"},
and many others in the biosciences domain. To be honest, for most large
datasets repositories already exist. The problem is not â€˜big
scienceâ€™, the problem is â€˜small scienceâ€™ or â€˜long tailâ€™
science. The vast quantity of data and description that is generated by
small numbers o people in isolated labs. Huge amounts of data
languishing on millions of laptops in non-standard formats. All that
stuff that no-one can really be bothered dealing with.

The storage itself is not really a problem. It can go on [Amazon
S3](http://en.wikipedia.org/wiki/Amazon_S3 "Amazon S3"){rel="wikipedia"},
Google servers, or into the various [Institutional
Repositories](http://en.wikipedia.org/wiki/Institutional_repository "Institutional repository"){rel="wikipedia"}
that Universities are rapidly deploying internationally. In fact, it
should go into all three and a few other places besides. Storage is
cheap, redoing the experiment is not. The digital curation people are
also working on the format problem with standards like
[ORE](http://www.ukoln.ac.uk/repositories/digirep/index/OAI-ORE)
providing a means of wrapping collections of data up and describing the
format internally. Combine this with an open repository of data formats
and the problem of storing and understanding data goes away, or rather
it is still a large technical problem but one that is reasonably soluble
with sufficient effort. One thing that is needed is standards for
describing data formats; both standards in terms of the description and
quality standards for what are the minimal requirements for describing
data (MiSDaFD anyone?). A lot of work is required in the area of
controlled vocabularies and ontologies, particularly in science domains
where they havenâ€™t yet got much traction. These donâ€™t need to be
perfect, they just need to be descriptive. But once again, how to go
about doing this is reasonably clear (devil is in the details I know).

The fundamental problem is how to capture the data, associated metadata,
and do the wrapping in a way that is an easy and convenient enough that
scientists will actually do it. It will come as no suprise to the
regularn readers of this blog that I would advocate the development of
tools to enable scientists to do this effectively at the stage at which
they already record their experiments; when they are done. Not only is
this the point where the scientist already has the task of planning,
executing, and recording their experiment. It is also the point at which
embedding good metadata and recording practice will be the most benefit
to them.

So we need to build a lab book system, or a virtual research environment
if you prefer, because this will encompass much more than the
traditional paper notebook. It will also encompass much more than the
current crop of electronic lab notebooks. I donâ€™t want to go into the
details of what I think this should look like here; thatâ€™s the subject
for a post in its own right. But to give an outline this system will
consist of a feed aggregator (not unlike
[Friendfeed](http://friendfeed.com)) where the user can subscribe to
various data streams, some from people, some from instruments or
sensors, and some that they have generated themselves, pictures at
[Flickr](http://flickr.com), comments on [Twitter](http://twitter.com).
The lab book component itself will look more or less like a word
processor, or a blog, or a wiki, depending on what the user wants. In
this system the user will generate documents (or pages); some will
describe samples, some procedures, and some ideas. Critically this
authoring system will have an open plugin architecture supported by an
open source community that enables the user to grab extra functionality
automatically.

A simple plugin might recognise a DNA sequence, or an INCHI code, and
provide a functional graphical display, complete with automatically
generated links back to the appropriate databases. Another plugin might
provide access to a range of ontologies or controlled vocabularies which
enable autocompletion of typed text with the appropriate name or easy or
automatically generate the PDB/SBML/MiBBI compliant datafiles required
for deposition. Journal submission templates could also be plugins or
gadgets that assist in pulling all the relevant information together for
submission and wrapping it up in an appropriate format. Semantic
authoring at the journal submission stage is never going to work.
Building it in early on and sucking it through when required just might.
Adding semantic information could be as easy (but probably should be no
easier than) formatting text. Select, right click, add semantics from
drop down menu.

This environment will have a proper versioning system, web integration
to push documents into the wider world, and a WYSIWYG interface. It will
expose human readable and machine readable versions of the files, and
will almost certainly use an underlying XML document format. It will
probably expose both microformats, rdf, and free text tags.

I think there are essentially two routes towards this kind of system.
One is to build into existing word processor and document handling
ecosystems. Essentially building a plugin architecture for Word or
OpenOffice. This would have the advantage of a system that is already
widely used and familiar to scientists. It has the disadvantage (at
least in the Microsoft case) that it would be built on top of a closed
source product and be reliant on the continued existence and support of
a large company. The alternative approach is to bring a completely open
source product through. This will require much more work, particularly
on the user interface side, but may be easier to handle in terms of open
source plugin architecture. It could be entirely web based. Google
AppEngine provides a very interesting environment for building such a
thing. CouchDB may be a good way of maintaining the underlying data.
Google Gadgets may well provide a lot of the desired plugin architecture
and widgets of various, types, javascript, bookmarklets, and all the
various tools on the web may provide a lot of the rest.

The purpose of this system is ultimately to encourage people to place
research results in the public domain in a way that makes them useful.
For it to get traction it has to be a no-brainer for people to adopt as
a central part of the way they run their laboratory. If it doesnâ€™t
make peopleâ€™s life easier; a lot easier; its not going to get any
traction. The User interface will be crucial, reliability will be
crucial, and the availability of added functionality which is currently
simply unavailable will be crucial.

Agreeing an architecture is likely to be a challenge but an open source
project is probably a more effective way of leveraging effort from a
wide community. There are many open standards available that can be used
here and the key is getting them all working together. Ideally the
choice of authoring environment wonâ€™t matter because it can still push
documents to the web in the same format(s), still use the same plugins,
still interface with the databases. Building this will cost money, quite
a lot of money, but the flip side is enabling the availability of the
data, the procedures into the public domain. As I heard Christoph
Steinbeck say yesterday, we can either get on and do this, or wait for
the singularity. I think weâ€™ve got a while to wait for that, so I
think its time to get on with the build. Who wants to be involved?

Related articles

- [The Open Data licensing
  issue](http://mndoci.com/blog/2008/05/11/the-open-data-licensing-issue/ "Open in new window")
  \[via Zemanta\]
- [Google App Engine Launches: Goes Directly After
  Amazon](http://www.centernetworks.com/google-app-engine-launches "Open in new window")
  \[via Zemanta\]
- [Reaching for the Sky Through The Compute
  Clouds](http://www.readwriteweb.com/archives/reaching_for_the_sky_through_compute_clouds.php "Open in new window")
  \[via Zemanta\]

[![](http://img.zemanta.com/pixie.png?x-id=7668403a-d9c4-4f53-b255-2fa59d7e22ec){decoding="async"}](http://www.zemanta.com/ "Zemified by Zemanta")[![](http://img.zemanta.com/pixie.png?x-id=){decoding="async"}](http://www.zemanta.com/ "Zemified by Zemanta")[![](http://img.zemanta.com/pixie.png?x-id=){decoding="async"}](http://www.zemanta.com/ "Zemified by Zemanta")

[![](http://img.zemanta.com/pixie.png?x-id=){decoding="async"}](http://www.zemanta.com/ "Zemified by Zemanta")


I realised the other day that I havenâ€™t written an exciteable blog post about getting an invitation to SciFoo! The reason for this is that I got overexcited over on FriendFeed instead and havenâ€™t really had time to get my head together to write something here.

Messaggi di Rogue Scholar

Taxonomy on a hard disk

How do we build the science data commons? A proposal for a SciFoo session