Björn Brembs

tl;dr: So far, I can't see any principal difference between our three
kinds of intellectual output: software, data and texts.

 

I admit I'm somewhat surprised that there appears to be a need to write
this post in 2014. After all, this is not really the dawn of the digital
age any more. Be that as it may, it is now March 6, 2014, six days since
PLoS's 'revolutionary' [data sharing
policy](https://www.plos.org/data-access-for-the-open-access-literature-ploss-data-policy/)
was revealed and only few people seem to observe the irony of avid
social media participants pretending it's still 1982. For the
uninitiated, just skim Twitter's
[\#PLoSfail](https://twitter.com/search?q=%23PLOSfail&src=tyah), read
[Edmund Hart's post](https://emhart.info/blog/2014/03/04/data-sharing/)
or see [Terry McGlynn's
post](https://smallpondscience.com/2014/03/03/i-own-my-data-until-i-dont/)
for some examples. I'll try to refrain from reiterating any arguments
made there already.

Colloquially speaking, one could describe the scientific method somewhat
shoddily as making an observation, making sense of the observation and
presenting the outcomes to interested audiences in some version of
language. Since the development of the scientific method somewhere
between the 16th century and now, this is roughly  how science has
progressed. Before the digital age,it was relatively difficult to let
everybody who was interested participate in the observations. Today,
this is much easier. It still varies tremendously between fields,
obviously, but compared to before, it's a world's difference. Today, you
could say that scientists collect data, evaluate the data and then
present the result in a scientific paper.

Data collection can either be done by hand or more or less
automatically. It may take years to sample wildlife in the rainforest
and minutes to evaluate the data on a spreadsheet. It may take decades
to develop a method and then seconds to collect the data. It may take a
few hours to generate the data first by hand and then by automated
processing, but decades before someone else comes up with the right way
to analyze and evaluate the data. What all scientific processes today
have in common is that at some point, the data becomes digitized, either
by commercial software or by code written precisely or that purpose.
Perhaps not in all, but surely in the vast majority of quantitative
sciences, the data is then evaluated using either commercial or
hand-coded software, be it for statistical testing, visualization,
modeling or parameter/feature extraction, etc. Only after all this is
done and understood does someone sit down and attempts to put the
outcome of this process into words that scientists not involved in the
work can make sense of.

Until about a quarter of a century ago, essentially all that was left of
the scientific process above were some instruments used to make the
observations and the text accounts of them. Ok, maybe some paper records
and later photographs. With a delay of about 25 years, the scientific
community is now slowly awakening to the realization that the
digitization of science would actually allow us to preserve the
scientific process much more comprehensively. Besides being a boon for
historians, reviewers, committees investigating scientific misconduct or
the critical public, preserving this process promises the added benefit
of being able to reward not only those few whose marketing skills
surpass the average enough to manage publishing their texts in glamorous
magazines, but also those who excel at scientific coding or data
collection. For the first time in human history, we may even have a shot
at starting to think about developing software agents that can trawl
data for testable hypotheses no human could ever come up with -- proofs
of concepts already exist. There is even the potential to [alert
colleagues](https://bjoern.brembs.net/2013/05/a-wonderful-example-of-open-science/)
to problems with their data, use the code for purposes the original
author did not dream of or extract parameters from data the
experimentalist had not the skill to do. In short, the advantages are
too many to list and reach far beyond science itself.

Much like the after the previous [hypothetical requirement of proofs for
mathematical
theorems](https://proteinsandwavefunctions.blogspot.dk/2014/03/top-10-reasons-to-not-share-your-data.html),
or the [equally hypothetical requirement of statistics and sound
methods](https://www.nature.com/news/announcement-reducing-our-irreproducibility-1.12852),
there is again resistance from the more conservative sections of the
scientific community for largely the [same 10
reasons](https://proteinsandwavefunctions.blogspot.dk/2014/03/top-10-reasons-to-not-share-your-data.html),
subsumed by: "it's too much work and it's against my own interests".

I can sympathize with this general objection as making code and data
available is more work and does facilitate scooping. However, the same
can be said of publishing traditional texts: it is a lot of work that
takes time away from experiments and opens all the flood gates of others
making a career on the back of your work. Thus, any consequential
proponent of "it's too much work and it's against my own interests"
ought to resist text publications with just as much fervor as they
resist publishing data and code. The same arguments apply.

Such principles aside, in practice, of course our general infrastructure
makes it much too difficult to publish either text, data or software,
which is so many of us now spend so much time and effort on publishing
reform and why our lab in particular is [developing ways to improve this
infrastructure](https://www.slideshare.net/brembs/making-science-open-by-default).
But that, as well, does also not differ between software, data and
science: our digital infrastructure is dysfunctional, period.

Neither does making your data and software available make you
particularly more liable to scooping or exploitation than the
publication of your texts does. The risks vary dramatically from field
to field and from person to person and are impossible to quantify.
Obviously, just as with text publications, data and code must be cited
appropriately.

There may be instances where the person writing the code or collecting
the data already knows what they want to do with the code/data next, but
this will of course take time and someone less gifted with ideas may be
on the hunt for an easy text publication. In such (rare?) cases, I think
it would be a practical solution to implement a limited provision on the
published data/code stating the precise nature of the planned research
and the time-frame within which it must be concluded. Because of its
digital nature, any violation of said provisions would be easily
detectable. The careers of our junior colleagues need to be protected
and any publishing policy on text, data or software ought to strive
towards maximizing such protections without hazarding the entire
scientific enterprise. Also here no difference between text,data and
software.

Finally, one might make a reasonable case that the rewards are stacked
disproportionately in favor of text publications, in particular with
regard to publications in certain journals. However, it
[almost](https://twitter.com/DNLee5/status/441511863122018304) goes
without saying that it is also unrealistic to expect tenure committees
and grant evaluators to assess software and data contributions before
anybody even is contributing and sharing data or code. Obviously, in
order to be able to reward coders and experimentalists just as we reward
the Glam hunters, we first need something to reward them for. That being
said, in today's antiquated system it is certainly a wise strategy to
prioritize Glam publications over code and data publications -- but
without preventing change for the better in the process. This is
obviously a chicken-and-egg situation which is not solved by the
involved parties waiting for each other. Change needs to happen on both
sides if any change is to happen.

To sum it up: our intellectual output today manifests itself in code,
data and text. All three are complementary and contribute equally to
science. All three expose our innermost thoughts and ideas to the
public, all three make us vulnerable to exploitation. All three require
diligence, time, work and frustration tolerance. All three constitute
the fruits of our labor, often our most cherished outcome of passion and
dedication. It is almost an insult to the coders and experimentalists
out there that these fruits should remain locked up and decay any
longer. At the very least, any opponent to code and data sharing ought
to consequentially also oppose text publications for exactly the same
reasons. We are already 25 years late to making our CVs contain code,
data and text sections under the "original research" heading. I see no
reason why we should be rewarding Glam-hunting marketers any longer.

**UPDATE**: I was just alerted to an important and relevant distinction
between text, data and code: [file
extension](https://twitter.com/IanMulvany/status/441651497257156608).
Herewith duly noted.


tl;dr: So far, I can’t see any principal difference between our three kinds of intellectual output: software, data and texts.   I admit I’m somewhat surprised that there appears to be a need to write this post in 2014. After all, this is not really the dawn of the digital age any more.

What is the difference between text, data and code?

University of Glasgow

Roderic Page

A while ago I posted [BHL to PDF
workflow](http://iphylo.blogspot.com/2012/06/bhl-to-pdf-workflow.html)
which was a sketch of a work flow to generate clean, searchable PDFs
from Biodiversity Heritage Library (BHL) content: 
 
![Workflow](http://lh6.ggpht.com/-KWASs15P3SM/T9thCN9SkxI/AAAAAAAABa4/ajJH8yZPzk8/workflow.png?imgmax=800 "workflow.png"){style="display:block; margin-left:auto; margin-right:auto;"
border="0" width="550" height="510"} 
I\'ve made some progress on putting this together, as well as expanded
the goal somewhat. In fact, there are several goals: 

1. BioStor articles need to be archived somewhere. At the moment they
 live on my server, and metadata is also served by BHL (as the
 \"parts\" you see in a scanned volume). Long term maybe PubMed
 Central is a possibility (BHL essentially becomes a publisher).
 Imagine PubMed Central becoming the primary archival repository for
 biodiversity literature.
2. BioStor articles could be more useful if the OCR text was cleaned up
 and marked up (e.g., highlighting taxon names, localities,
 extracting citations, etc.).
3. If BioStor articles were marked up to same extent as *ZooKeys* then
 we could use tools developed for *ZooKeys* (see [Towards an
 interactive taxonomic article: displaying an article from
 ZooKeys](http://iphylo.blogspot.com/2011/12/towards-interactive-taxonomic-article.html))
 for a richer reading experience.
4. Cleaned OCR text could also be used to generate searchable PDFs,
 which are still the most popular way for people to read articles
 (see [Why do scientists tend to prefer PDF documents over HTML when
 reading scientific
 journals?](http://www.quora.com/Why-do-scientists-tend-to-prefer-PDF-documents-over-HTML-when-reading-scientific-journals)).
 BioStor already [generates
 PDFs](http://iphylo.blogspot.com/2010/04/biostor-gets-pdfs-with-xmp-metadata.html),
 but these are simply made by wrapping page images in a PDF.
 Searchable PDFs would be much friendlier.

 
For BioStor articles to be archived in PubMed Central they would need to
be marked up using the [Journal Archiving and Interchange Tag
Suite](http://jats.nlm.nih.gov) (formerly the NLM DTDs). This is the
markup used by many publishers, and also the tag suite that
[TaxPub](http://www.ncbi.nlm.nih.gov/books/NBK100351/) build upon. 
 
The idea of having BioStor marked up in JATS is appealing, but on the
face of it impossible because the all we have is page scans and some
[pretty ropey OCR](http://iphylo.blogspot.com/2010/12/bhl-and-ocr.html).
But because the NLM has also been heavily involed in scanning the
historical literature they are used to dealing with scanned literature,
and JATS can accommodate articles ranging from scans to fully marked up
text. For example, take a look at the article \"Microsporidian
encephalitis of farmed Atlantic salmon (Salmo salar) in British
Columbia\" which is in PubMed Central
([PMC1687123](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1687123/)).
PMC has basic metadata for the article, scans of the pages, and two
images extracted from those pages. This is pretty much what BioStor
already has (minus the extracted images). 
 
With this in mind, I dusted off some old code, put it together and
created an example of the first baby steps towards BioStor and JATS. The
code is in [github](https://github.com/rdmpage/biostor-jats){<="" a=""},
and there is a live example
[here](https://dl.dropboxusercontent.com/u/639486/65706/65706.html). 
 
![Jats](http://lh4.ggpht.com/-eSgZZX6ZNts/Up83my8dReI/AAAAAAAADMw/YJMSXlgipLU/jats.png?imgmax=800 "jats.png"){style="display:block; margin-left:auto; margin-right:auto;"
border="0" width="550"} 
The example takes BioStor article
[65706](http://biostor.org/reference/65706), converts the metadata to
JATS, links in the page scans, and also extracts images from the page
scans based on data in the ABBYY OCR files. I\'ve also generated HTML
from the DjVu files, and this HTML includes hOCR tags that embed
information about the OCR text. This format can be edited by tools such
as (see Jim Garrison\'s moz-hocr-edit discussed in [Correcting OCR using
hOCR in
Firefox](http://iphylo.blogspot.com/2011/07/correcting-ocr-using-hocr-firefox.html)).
This HTML can be processed to output a PDF that includes the page scans
but also has the OCR text as \"hidden text\" so the reader can search
for phrases, or copy and paste the text (try the [PDF for article
65706](https://dl.dropboxusercontent.com/u/639486/65706/65706.pdf)). 
 
I\'ve put the HTML (and all the XML and images) in github, so one
obvious model for working on an article is to put it into a github
repository, push any edits made to the repository, then push that to a
web server that displays the articles. 
 
There are still a lot of rough edges, and I think we can buld nicer
interfaces than moz-hocr-edit (e.g., using the \"contenteditable\"
attribute in the HTML), althogh moz-hocr-edit has the nice feature of
being able to save the edits straight back to the HTML file (saving
edited HTML to disk is a non-trivial task in most web browsers). I also
need to add the code for building the initial JATS file (currently this
is hidden on the BioStor server). There are also issues about PDF
quality. At the moment I output black and white PNGs, which look nice
and clean but can mangle plates and photos. I need to tweak that aspect
of the process. 
 
One application of these tools would be to take a single journal and
convert all the BioStor articles into JATS, then make it available for
people to further clean and markup as needed. There is an extraordinary
amount of information locked away in this literature, it would be nice
if we made better use of that treasure trove.

A while ago I posted BHL to PDF workflow which was a sketch of a work flow to generate clean, searchable PDFs from Biodiversity Heritage Library (BHL) content:   I've made some progress on putting this together, as well as expanded the goal somewhat. In fact, there are several goals: BioStor articles need to be archived somewhere.

Towards BioStor articles marked up using Journal Archiving Tag Set

Quick note to self about exporting data from my [Australian Faunal
Directory on CouchDB](http://iphylo.org/~rpage/afd/) project. To export
data from a CouchDB view you can use a list function (see [Formatting
with Show and
List](http://wiki.apache.org/couchdb/Formatting_with_Show_and_List)).
Following the example on the [Kanapes
IDE](http://blog.kanapeside.com/csv-export-via-couchdb-list-function)
blog, I created the following list function: 
 

``` {style="font-size:10px;border:1px solid rgb(192,192,192);background-color:rgb(240,240,240);"}
{
"_id": "_design/publication",
"_rev": "14-467dee8248e97d874f1141411f536848",
"language": "javascript",
"lists": {
"tsv": "function(head,req) {
var row;
start({
'headers': {
'Content-Type': 'text/tsv'
}
});
while(row = getRow()) {
send(row.value + '\\t' + row.key + '\\n');
}}"
},
"views": {
.
.
.
}
}
```

 
 
I can use this function with the view below, which lists Australian
Faunal Directory publications by UUID (\"value\"), indexed by DOI
(\"key\"). 
 
![Couch](http://lh6.ggpht.com/-wsNeSZucCJI/TuX8WtDuW3I/AAAAAAAABGg/ulfCL-PjDuY/couch.png?imgmax=800){style="display:block; margin-left:auto; margin-right:auto;"
border="0" width="400"} 
 
I can get the tab-delimited dump from
<http://localhost:5984/afd/_design/publication/_list/tsv/doi>. Note that
instead of, say, `/afd/_design/publication/_view/doi` to get the view,
we use `/afd/_design/publication/`**`_list/tsv/`**`doi` to get the
tab-delimited dump. 
 
I\'ve created files listing
[DOIs](http://iphylo.org/~rpage/afd/downloads/doi.txt) and
[BioStor](http://iphylo.org/~rpage/afd/downloads/biostor.txt) ids for
publications in the Australian Faunal Directory. I\'ll play with lists a
bit more, specially as I would like to extract the mapping from the
[Australian Faunal Directory on CouchDB](http://iphylo.org/~rpage/afd/)
project and add it to the [iTaxon](http://iphylo.org/~rpage/itaxon/)
project.

Quick note to self about exporting data from my Australian Faunal Directory on CouchDB project. To export data from a CouchDB view you can use a list function (see Formatting with Show and List). Following the example on the Kanapes IDE blog, I created the following list function: { "_id": "_design/publication", "_rev": "14-467dee8248e97d874f1141411f536848", "language": "javascript", "lists": { "tsv": "function(head,req) { var row;

Exporting data from Australian Faunal Directory on CouchDB

![mendeley.png](http://lh3.ggpht.com/_Gct8lVAxKqQ/TGOqbgGhEbI/AAAAAAAAAtM/c63RwupveNg/mendeley.png?imgmax=800){border="0"
width="128" height="128" align="right"}Following on from my [earlier
post](http://iphylo.blogspot.com/2010/08/more-on-mendeley-api.html)
about the Mendeley API, I\'ve bundled up my code for OAuth access to the
Mendeley API for anyone who\'s interested in playing with the API using
PHP. You can [browse the
code](http://code.google.com/p/iphylo/source/browse/#svn/trunk/mendeley-oapi-php)
on Google Code, or grab a [tarball
here](http://code.google.com/p/iphylo/downloads/detail?name=mendeley-oapi-php.tgz).
You\'ll need a consumer key and a consumer secret from Mendeley for the
demos to work, and if you\'re behind a HTTP proxy you\'ll have to tweak
the code (this is explained in the ReadMe.txt file that comes with the
code).  
  
The code is pretty rough, and doesn\'t use all the Mendeley API calls,
but I\'ve other things to do, and it felt like a case of either bundle
this up now, or it will get lost among a host of other projects. The
Mendeley API still feels woefully under-developed. I\'d be more
interested in developing this client further if the API was powerful
enough to do the kinds of things I\'d like to do.


Following on from my earlier post about the Mendeley API, I've bundled up my code for OAuth access to the Mendeley API for anyone who's interested in playing with the API using PHP. You can browse the code on Google Code, or grab a tarball here.

Mendeley API PHP client

Following on from [the previous
post](http://iphylo.blogspot.com/2008/10/books.html), I wrote a simpe
Mediawiki extension to insert a Google Book into a wiki page. Written in
a few minutes, not tested much, etc. 
 
To use this, copy the code below and save in a file **googlebook.php**
in the **extensions** directory of your Mediawiki installation. 
 

``` {style="border: 1px solid rgb(199, 207, 213);margin: 20px 0pt; padding: 15px; background: rgb(241, 245, 249)"}
# rdmp

# Google Book extension
# Embed a Google Book into Mediawiki
#
# Usage:
# 
# 
# To install it put this file in the extensions directory 
# To activate the extension, include it from your LocalSettings.php
# with: require("extensions/googlebook.php");
 
$wgExtensionFunctions[] = "wfGoogleBook";
 
function wfGoogleBook() {
 global $wgParser;
 # registers the extension with the WikiText parser
 $wgParser->setHook( "googlebook", "renderGoogleBook" );
}
 
# The callback function for converting the input text to HTML output
function renderGoogleBook( $input, $argv ) 
{
 $width = 425;
 $height = 400;
 
 if (isset($argv["width"]))
 {
 $width = $argv["width"];
 }
 if (isset($argv["height"]))
 {
 $width = $argv["height"];
 }
 
 $output = ' src="http://books.google.com/books/previewlib.js">
 ';
 $output .= ' GBS_insertEmbeddedViewer(\'' . $argv["id"] . '\','. $width . ',' . $height . '); '; 

 return $output;
}
?>
```

 
 
In your **LocalSettings.php** file add the line 
 

``` {style="border: 1px solid rgb(199, 207, 213);margin: 20px 0pt; padding: 15px; background: rgb(241, 245, 249)"}
require("extensions/googlebook.php");
```

 
 
Now you can add a Google book to a wiki page by adding a tag. For
example: 
 

``` {style="border: 1px solid rgb(199, 207, 213);margin: 20px 0pt; padding: 15px; background: rgb(241, 245, 249)"}
```

 
 
The **id** gives the book identifier (such as an OCLC number or a ISBN
(you need to include the identifier prefix). By defaulot, the book will
appear in a box 425 × 400 pixels in size. You can add optional **width**
and **height** parameters to adjust this.

Following on from the previous post, I wrote a simpe Mediawiki extension to insert a Google Book into a wiki page. Written in a few minutes, not tested much, etc. To use this, copy the code below and save in a file

 googlebook.php

in the

 extensions

directory of your Mediawiki installation.

Messaggi di Rogue Scholar

What is the difference between text, data and code?

Towards BioStor articles marked up using Journal Archiving Tag Set

Exporting data from Australian Faunal Directory on CouchDB

Mendeley API PHP client

Google Books and Mediawiki