University of Glasgow

Roderic Page

![Refine](http://lh4.ggpht.com/-HsS8f5sKWWc/Ty_fgRiXiII/AAAAAAAABHg/In4r9NXMkoc/refine.png?imgmax=800){border="0"
width="128" height="128" style="float:right;"}In an [earlier
post](http://iphylo.blogspot.com/2012/02/using-google-refine-and-taxonomic.html)
I discussed using [Open Refine](http://openrefine.org/) (formerly
[Google Refine](https://code.google.com/p/google-refine/)) to clean and
reconcile taxon names. I\'ve added an additional service that can be
used to reconcile author names that uses the [Virtual International
Authority File (VIAF)](http://viaf.org/) API. Using this service we can
match authors to VIAF identifiers (you may have noticed these appearing
on people\'s pages in Wikipedia, e.g. [Mary J.
Rathbun](http://en.wikipedia.org/wiki/Mary_Rathbun)\'s Wikipedia page
lists her VIAF as [61796012](http://viaf.org/viaf/61796012)).  
  
To use the service follow the instructions in the [earlier
post](http://iphylo.blogspot.com/2012/02/using-google-refine-and-taxonomic.html)
but add the service:  
  
http://iphylo.org/\~rpage/phyloinformatics/services/reconciliation_viaf.php  
  
This service is fairly crude, in particular, I make no attempt to score
the matches that VIAR returns because this would require parsing and
normalising author names. This could be added if needed. If you want
some exmaple names to try, here are some taxonomists:  
  

    George A Boulenger
    G A Boulenger
    Wilhelm Michaelsen
    W Michaelsen
    Colin Campbell Sanborn
    Suzanne Hand
    Philip Hershkovitz
    Yehudah Leopold Werner
    W B Spencer
    Norman Platnick


In an earlier post I discussed using Open Refine (formerly Google Refine) to clean and reconcile taxon names. I've added an additional service that can be used to reconcile author names that uses the Virtual International Authority File (VIAF) API.

Reconciling author names using Open Refine and VIAF

![Refine](http://lh4.ggpht.com/-HsS8f5sKWWc/Ty_fgRiXiII/AAAAAAAABHg/In4r9NXMkoc/refine.png?imgmax=800){border="0"
width="128" height="128" style="float:right;"}[Google
Refine](http://code.google.com/p/google-refine/) is an elegant tool for
data cleaning. One of its most powerful features is the ability to call
\"Reconciliation Services\" to help clean data, for example by matching
names to external identifiers. Google Refine comes with the ability to
use [Freebase](http://www.freebase.com/) reconciliation services, but
you can also add external services. Inspired by this I\'ve started to
implement services to reconcile taxonomic names.  
  
The services I\'ve implemented so far are:  

- [EOL](http://www.eol.org)
  http://iphylo.org/\~rpage/phyloinformatics/services/reconciliation_eol.php
- [NCBI taxonomy](http://www.ncbi.nlm.nih.gov/Taxonomy/)
  http://iphylo.org/\~rpage/phyloinformatics/services/reconciliation_ncbi.php
- [uBio FindIT](http://www.ubio.org/tools/recognize.php)
  http://iphylo.org/\~rpage/phyloinformatics/services/reconciliation_ubio.php
- [WORMS](http://www.marinespecies.org/)
  http://iphylo.org/\~rpage/phyloinformatics/services/reconciliation_worms.php
- [GBIF](http://www.gbif.org)
  http://iphylo.org/\~rpage/phyloinformatics/services/reconciliation_gbif.php
- [Global Names Index](http://gni.globalnames.org/)
  http://iphylo.org/\~rpage/phyloinformatics/services/reconciliation_globalnames.php

  
  
To use these you need to add the URLs above to Google Refine (see
example below). The EOL, NCBI and WORMS do a basic name lookup. The uBio
FindIT service extracts a taxonomic name from a string, and can be
viewed as a \"taxonomic name cleaner\".  
  
**How to use reconciliation services**  
  
Start a Google Refine session. Save the names below to a text file and
open it as a new project.  

    Names
    Achatina fulica (giant African snail)
    Acromyrmex octospinosus ST040116-01
    Alepocephalus bairdii (Baird's smooth-head)
    Alaska Sea otter (Enhydra lutris kenyoni)
    Toxoplasma gondii
    Leucoagaricus gongylophorus
    Pinnotheres
    Themisto gaudichaudii
    Hyperiidae

  
  
You should see something like this:  
![Refine1](http://lh5.ggpht.com/-PqIRaUwZy40/Ty_fhvgf87I/AAAAAAAABHo/ZbMluFovHYc/refine1.png?imgmax=800){style="display:block; margin-left:auto; margin-right:auto;"
border="0" width="359" height="311"}  
  
Click on the column header **Names** and choose **Reconcile** → **Start
reconciling**.  
  
![Refine2](http://lh6.ggpht.com/-rDs2S-YG-V8/Ty_fiitowGI/AAAAAAAABHw/rJ3Uf6RcVQ8/refine2.png?imgmax=800){style="display:block; margin-left:auto; margin-right:auto;"
border="0" width="415" height="436"}  
  
A dialog will popup asking you to select a service.  
  
![Refine3](http://lh4.ggpht.com/-iMMUwS6tmc8/Ty_fjxRZnRI/AAAAAAAABH4/rLVPLLyDVKs/refine3.png?imgmax=800){style="display:block; margin-left:auto; margin-right:auto;"
border="0" width="415" height="285"}  
  
If you\'ve already added a service it will be in the list on the left.
If not, click the **Add Standard Services\...** button at the bottom
left and paste in the URL (in this case
`http://iphylo.org/~rpage/phyloinformatics/services/reconciliation_ubio.php`).  
  
Once the service has loaded click on **Start Reconciling**. Once it has
finished you should see most of the names linked to uBio (click on a
name to check this):  
  
![Refine4](http://lh6.ggpht.com/-hG9-5E4YaDY/Ty_fkxZnckI/AAAAAAAABIA/IaAbrXWzPv4/refine4.png?imgmax=800){style="display:block; margin-left:auto; margin-right:auto;"
border="0" width="343" height="438"}  
  
Sometimes there may be more than one possible match, in which case these
will be listed in the cell. Once you have reconciled the data you may
want to do something with the reconciliation. For example, if you want
to get the ids for the names you\'ve just matched you can create a new
column based on the reconciliation. Click on the **Names** column header
and choose **Edit column** → **Add column based on this column\...**. A
dialog box will be displayed:  
  
![Refine6](http://lh6.ggpht.com/-T9cbG-OY8KA/Ty_fmKBujbI/AAAAAAAABII/R-rkj8Lj3w4/refine6.png?imgmax=800){style="display:block; margin-left:auto; margin-right:auto;"
border="0" width="415" height="307"}  
  
In the box labelled **Expression** enter `cell.recon.match.id` and give
the column a name (e.g., \"NamebankID\"). You will now have a column of
uBio NamebankIDs for the names:  
  
![Refine7](http://lh6.ggpht.com/-5gd07QAHzWY/Ty_fnobAUtI/AAAAAAAABIQ/fx6W_iTy-B8/refine7.png?imgmax=800){style="display:block; margin-left:auto; margin-right:auto;"
border="0" width="378" height="439"}  
  
You could also get the names uBio extracted by creating a column based
on the values of `cell.recon.match.name`. To compare this with the
original values, click on the **Names** column header and choose
**Reconcile** → **Actions** → **Clear reconciliation data**. Now you can
see the original input names, and the string uBio extracted from each
name:  
  
![Refine8](http://lh5.ggpht.com/-WfFkd_MxqXc/Ty_foWnMx2I/AAAAAAAABIY/7NYNqO0qx7o/refine8.png?imgmax=800){style="display:block; margin-left:auto; margin-right:auto;"
border="0" width="415" height="217"}  
  
These are some very simple ideas for using Google Refine with taxonomic
name services. Obvious extensions would to use services that provide an
\"accepted name\", or services that support approximate string matching
so you could catch spelling mistakes (most of the services I\'ve
implemented here have some degree of support for these features).  
  
**Development notes**  
The code for these services is in
[Github](https://github.com/rdmpage/phyloinformatics) (undocumented as
yet, that\'s on the to do list). I had a few hiccups getting these
services to work. There is detailed documentation at
<http://code.google.com/p/google-refine/wiki/ReconciliationServiceApi>,
but this seems a little out of step with what actually happens. Based on
the documentation I thought Google Refine called a reconciliation
service using HTTP GET, but in fact it uses POST. Google Refine always
called my reconciliation service using \"Multiple Query Mode\", which
meant supporting this mode wasn\'t optional. Once these issues were
sorted out (turning on the Java console as per [David Huynh\'s
tip](https://groups.google.com/forum/?fromgroups#!topic/google-refine/mdUMgaf7ntY)
helped) things work pretty well.


Google Refine is an elegant tool for data cleaning. One of its most powerful features is the ability to call "Reconciliation Services" to help clean data, for example by matching names to external identifiers. Google Refine comes with the ability to use Freebase reconciliation services, but you can also add external services. Inspired by this I've started to implement services to reconcile taxonomic names.

Messaggi di Rogue Scholar

Reconciling author names using Open Refine and VIAF

Using Google Refine and taxonomic databases (EOL, NCBI, uBio, WORMS) to clean messy data