University of Glasgow

Roderic Page

In the [previous
post](http://iphylo.blogspot.com/2009/06/eol-wikipedia-tdwg-linkeddata-and.html)
I suggested that a productive way to meet EOL\'s goal of a web page per
taxon would be to build upon Wikipedia, rather than go it alone. In a
nutshell the arguments were:  

1.  Wikipedia has considerable traction and has some richly populated
    taxon pages
2.  The linked data community uses DBPedia.org as a core source of URIs
    for entities, as as DBPedia is derived from Wikipedia the later will
    be the core source of identifiers for taxa

To explore this a little further I grabbed two files from the [20090618
Wikipedia dump](http://download.wikimedia.org/enwiki/20090618/), namely
**page.sql** and **templatelinks.sql**, and extracted page ids and
titles for Wikipedia pages containing the [Taxobox
template](http://en.wikipedia.org/wiki/Template:Taxobox). I then queried
Wikipedia for the source for each of these pages, and tried to extract
the taxonomic information from each page (a tedious and error-prone
process at best).  
  
I\'ve put together a [shockingly crude web page where you can browse the
results](http://bioguid.info/demos/wikipedia/) (warning, this page is a
10 minute hack with little error checking).  
  
There is some good news. There are over 120,000 taxon pages (I\'ve not
got an exact figure because the Taxobox template occurs on some pages
that aren\'t taxon pages, such as documentation and user pages). Some
pages are extensive (the largest page is
[Dinosaur](http://en.wikipedia.org/wiki/Dinosaur) for which the source
text is 128K in size), and there are lots of links to external
references (I counted 7205 distinct DOIs to papers and/or books, and
3248 distinct ISBNs). This represents a degree of external linkage that
puts EOL to shame.  
  
However, there are also some major problems. Firstly, Wikipedia does not
have a single, internally consistent classification (i.e., the
classification is not a tree). This is not unexpected, given that
Wikipedia pages comprise semi-structured text that is (largely) manually
entered. It\'s not a database. If it were, the simplest way to ensure
consistency would be to have each child node include a pointer to its
parent, and when we want a list of the children of the parent node we
simply query the database (\"what nodes have this node as their
parent?\"). Because Wikipedia isn\'t a database, authors have entered
these two relationships (\"has parent\" and \"has child\") on different
pages, and these often conflict. For a spectacular example of this, take
a look at the page for
[Amphibia](http://bioguid.info/demos/wikipedia/?title=Amphibia). When I
scrapped Wikipedia I extracted the \"has parent\" link, as this is the
simplest way to create a tree. This results in over 200 child taxa for
Amphibia, yet the Wikipedia page for Amphibia lists only four child
taxa. What appears to be happening is that many fossil taxa are being
added to Wikipedia, and since we are often hazy about where they go in
the tree, authors are listing their parent taxon as (in this case)
\"Amphibia\". Given this direct link, they should also be listed as
children of Amphibia (although, of course, that would make a mess of the
Amphibia page). Perhaps the solution is to add a \"*incerta sedis*\"
taxon page for each taxon, and make that the parent of all the taxa that
we\'re aren\'t sure where to put. This would ensure consistency, but not
make the current taxon pages unreadable.  
  
Homonymy (the same name for different taxa) also raises it\'s ugly head.
For example, the page for the crab family
[Latreilliidae](http://en.wikipedia.org/wiki/Latreilliidae) lists the
genus [*Latreillia*](http://en.wikipedia.org/wiki/Latreillia), which is
a fly. In this case, the fly genus *Latreillia* Robineau-Desvoidy, is a
junior homonym of the crab genus *Latreillia* Roux (see
<http://biodiversitylibrary.org/page/12221111>).  
  
Finally, the page titles (which become the basis of DBPedia.org URIs)
are a muddled mixture of common and scientific names.  
  
So, what to do? Well, the idea of simply using Wikipedia as is isn\'t
going to fly, it\'s too broken. We will have to contemplate a concerted
effort to fix it (which will require using bots to clean up the
inconsistencies). Another option (assuming that we like the Wiki-style
environment) is to use a semantic wiki (see [my earlier
post](http://iphylo.blogspot.com/2009/02/wiki-demo.html)), which
constrains some of the possible markup, but retains a lot of the freedom
that make wikis so powerful.  
  
This isn\'t an argument for not using Wikipedia as such, it\'s arguably
still much more informative than, say, EOL. It\'s just that it\'s
showing signs of the limitations of free-form text entry. The trick is
to find a way to combine the obvious strengths of this approach (ease of
creating and editing pages, massive community support) with the more
structured approach needed to avoid the internal inconsistencies that
currently bedevil Wikipedia.


In the previous post I suggested that a productive way to meet EOL's goal of a web page per taxon would be to build upon Wikipedia, rather than go it alone.

Wikipedia taxonomy, the good, the bad, and the very ugly

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWHom7WgHOZ1CoZfG41Vk9KGGX33TjGbE9-k7aEr31qr00HnjbFHRN6mlF91fRTkEQNzwzY1Rs2vYanE8Yzx-C0fkzxhHJ2PoLt7aorONEsgflJeFwGi9ZniZJ1RGcaq88XgZo/s320/ff_scraping_630.jpg){#BLOGGER_PHOTO_ID_5163422550296728898
style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;"
border="0"}](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWHom7WgHOZ1CoZfG41Vk9KGGX33TjGbE9-k7aEr31qr00HnjbFHRN6mlF91fRTkEQNzwzY1Rs2vYanE8Yzx-C0fkzxhHJ2PoLt7aorONEsgflJeFwGi9ZniZJ1RGcaq88XgZo/s1600-h/ff_scraping_630.jpg){onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"}  
Wired 16.01 has an article entitled [The Data
Wars](http://www.wired.com/techbiz/media/magazine/16-01/ff_scraping?currentPage=all)
by [Josh McHugh](http://stubble.blogs.com/joshmchugh/). A quote from the
printed version:  

> They call it scraping --- when web companies automatically harvest
> information from the likes of Yahoo, Google, and craigslist. Now the
> Internet establishment is clamping down.

  
It\'s a sobering read for those of us who advocate harvesting data from
as many sources as possible, more so in light of [Microsoft\'s bid to
buy
Yahoo](http://www.theregister.co.uk/2008/02/01/microsoft_bids_for_yahoo/).
Yahoo provides free access to many of its tools via an API (such as the
image search I use in [iSpecies](http://ispecies.org), and in this sense
is much more open than Google. Might this change under Microsoft\...?


Wired 16.01 has an article entitled The Data Wars by Josh McHugh. A quote from the printed version:  It's a sobering read for those of us who advocate harvesting data from as many sources as possible, more so in light of Microsoft's bid to buy Yahoo. Yahoo provides free access to many of its tools via an API (such as the image search I use in iSpecies, and in this sense is much more open than Google. Might this change under Microsoft...?

Messaggi di Rogue Scholar

Wikipedia taxonomy, the good, the bad, and the very ugly

The Data Wars