University of Glasgow

Roderic Page

I\'ve tweaked [Ozymandias](https://ozymandias-demo.herokuapp.com/) to
now include short natural language summaries (snippets) for various
taxa. This makes the output a little more friendly and informative. For
example, here\'s a snippet from the page on
[*Cephalodesmius*](https://ozymandias-demo.herokuapp.com/?uri=https://bie.ala.org.au/species/urn:lsid:biodiversity.org.au:afd.taxon:cc39fc0d-fc74-417d-b281-3804225d1b10),
a dung beetle that makes its own dung.  
  

::: {.separator style="clear: both; text-align: center;"}
[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieKVAQUO0pLh8ZJU5Ve5Wcdtl3Dgq1Crk-n_3zLN1lvL_ox6thuzMR3wncqWFw5jZR-Ec4zRMX7BxHI2G1kITPRAEuUNaxjugcC6i2d5Q9EkoZMSyQut-_e2Ivlq7GmPgw6szW/s400/Screenshot+2019-05-28+12.45.35.png){border="0"
original-height="682" original-width="1070" height="253"
width="400"}](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieKVAQUO0pLh8ZJU5Ve5Wcdtl3Dgq1Crk-n_3zLN1lvL_ox6thuzMR3wncqWFw5jZR-Ec4zRMX7BxHI2G1kITPRAEuUNaxjugcC6i2d5Q9EkoZMSyQut-_e2Ivlq7GmPgw6szW/s1600/Screenshot+2019-05-28+12.45.35.png){imageanchor="1"
style="margin-left: 1em; margin-right: 1em;"}
:::

  
These snippets come from Wikipedia, well actually, from the
[DBpedia](https://wiki.dbpedia.org/) project. Behind the scenes I have a
script that takes the GBIF taxon id for an ALA taxon (if it exists),
queries Wikidata for the corresponding taxon and any associate
identifiers of interest, and if there\'s a link to an English language
Wikipedia page I do a quick SPARQL query to DBpedia to retrieve the
snippet of text. At some point all of this could be sped up by adding
the relevant data to the triple store and doing the query locally but
for now it works well enough.  
  
Of course, many snippets are little more than stubs, e.g. the snippet
for another dung beetle genus
[*Diorygopyx*](https://ozymandias-demo.herokuapp.com/?uri=https://bie.ala.org.au/species/urn:lsid:biodiversity.org.au:afd.taxon:d1bac818-662a-4a11-99b3-c9c54f857f31)
doesn\'t tell us much more than we can get from the information already
displayed.  
  

::: {.separator style="clear: both; text-align: center;"}
[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZ-p_XBi6oK_qvc86Bea7Bl-30_vr1Oj14v7jFUqZos2k7KvcQpvY3asK1axFee7phSHtV146mABs0FQhhfPK-wDdPFHrxZecZuoCAbwK65KbMx2yqZ2C6x8CfvF7rbsO0sFw0/s400/Screenshot+2019-05-28+12.45.07.png){border="0"
original-height="278" original-width="1056" height="105"
width="400"}](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZ-p_XBi6oK_qvc86Bea7Bl-30_vr1Oj14v7jFUqZos2k7KvcQpvY3asK1axFee7phSHtV146mABs0FQhhfPK-wDdPFHrxZecZuoCAbwK65KbMx2yqZ2C6x8CfvF7rbsO0sFw0/s1600/Screenshot+2019-05-28+12.45.07.png){imageanchor="1"
style="margin-left: 1em; margin-right: 1em;"}
:::

  
  
But having a text summary still seems worthwhile, which raises the
question of what to do when Wikipedia doesn\'t know anything about a
taxon? Obviously, we could start editing Wikipedia to flesh out its
content, but that will take a while to filter into databases such as
DBpedia. Another approach is to generate snippets from the triple store
itself, in other words, generate [natural language summaries from
structured
data](https://en.wikipedia.org/wiki/Natural-language_generation). For
example, we could generate summaries such as \"Diorygopyx is a genus of
Scarabaeidae or scarab beetles in the superfamily Scarabaeoidea\" fairly
easily from knowing the taxonomic hierarchy and a few common names. But
we could also do more. In browsing Ozymandias I\'m struck at times by
how much our knowledge of one taxon depends on a major piece of
taxonomic work, often done some time ago. For example, [The Australian
Crickets (Orthoptera: Gryllidae). Academy of Natural Sciences of
Philadelphia, Monograph
22](https://ozymandias-demo.herokuapp.com/?uri=https://biodiversity.org.au/afd/publication/b12d8397-6321-4436-a028-178c318fb1b1)
by Otte and Alexander (1983) is a monumental taxonomic monograph, and
many Australian cricket genera had most (or all) of their species
described in that work. Imagine having a snippet that mentioned that
(e.g., \"Most species in this genus were described in 1983, no species
have been discovered since.\"). That would give the reader some useful
information, and perhaps also prompt them to ask \"so, why haven\'t any
more species been described?\".  
  
I think there\'s scope here to make the output from triple stores (and
other databases) more approachable using natural language generation.
This is obviously a big area, and there are some very sophisticated
approaches for outputting very natural language (think chatbots),
perhaps the most striking example of which is Google Duplex.  

::: {style="text-align: center;"}
  
:::

::: {style="text-align: center;"}
::: {.iframe}
::: {#player}
:::

::: {.player-unavailable}
# Ein Fehler ist aufgetreten. {#ein-fehler-ist-aufgetreten. .message}

::: {.submessage}
[Sieh dir dieses Video auf www.youtube.com
an](https://www.youtube.com/watch?v=D5VN56jQMWM){target="_blank"} oder
aktiviere JavaScript, falls es in deinem Browser deaktiviert sein
sollte.
:::
:::
:::
:::

  
But we don\'t need *quite* this level of sophistication, something using
much simpler techniques (e.g.,
[nalgene-js](https://github.com/spro/nalgene-js)) would probably be
enough. Armed with some basic facts from the triple store, and some
simple templates, we could probably generate some useful text snippets
for many taxa in Ozymandias, and indeed for other entities. For example,
David Shorthouse is outputting simple text summaries of the contribution
of taxonomists to specimen collection and identification:  

> ::: {dir="ltr" lang="en"}
> Arthur Loveridge identified Scolecomorphidae and collected
> Chamaeleonidae <https://t.co/NimKzmUuOX>
> :::
>
> --- Bloodhound (@BloodhoundTrack) [May 24,
> 2019](https://twitter.com/BloodhoundTrack/status/1131898443709014016?ref_src=twsrc%5Etfw)

  
Imagine extending this to take into account publications, geography,
etc. I think there\'s lots of scope here for moving beyond just
displaying data and trying to generate human-friendly summaries of data.


I've tweaked Ozymandias to now include short natural language summaries (snippets) for various taxa. This makes the output a little more friendly and informative. For example, here's a snippet from the page on
<em>
 Cephalodesmius
</em>
, a dung beetle that makes its own dung. These snippets come from Wikipedia, well actually, from the DBpedia project.

Ozymandias meets Wikipedia, with notes on natural language generation

Lincoln Mullen

The R package ecosystem for natural language processing has been
flourishing in recent days. R packages for text analysis have usually
been based on the classes provided by the
[NLP](https://cran.r-project.org/package=NLP/) or
[tm](https://cran.r-project.org/package=tm/) packages. Many of them
depend on Java. But recently there have been a number of new packages
for text analysis in R, most notably
[text2vec](https://github.com/dselivanov/text2vec),
[quanteda](https://github.com/kbenoit/quanteda), and
[tidytext](https://github.com/juliasilge/tidytext). These packages are
built on top of [Rcpp](http://www.rcpp.org/) instead of
[rJava](https://cran.r-project.org/package=rJava/), which makes them
much more reliable and portable. And instead of the classes based on
NLP, which I have never thought to be particularly idiomatic for R, they
use standard R data structures. The text2vec and quanteda packages both
rely on the sparse matrices provided by the rock solid
[Matrix](https://cran.r-project.org/package=Matrix/) package. The
tidytext package is idiosyncratic (in the best possible way!) for doing
all of its work in data frames rather than matrices, but a data frame is
about as standard as you can get. For a long time when I would recommend
R to people, I had to add the caveat that they should use Python if they
were primarily interested in text analysis. But now I no longer feel the
need to hedge.

Still there is a lot of duplicated effort between these packages on the
one hand and a lot of incompatibilities between the packages on the
other. The R ecosystem for text analysis is not exactly coherent or
consistent at the moment.

My small contribution to the new text analysis ecosystem is the
tokenizers package, which was recently accepted into rOpenSci after a
careful [peer
review](https://github.com/ropensci/software-review/issues/33) by [Kevin
Ushey](https://kevinushey.github.io/). A new version of the package is
[on CRAN](https://cran.r-project.org/package=tokenizers/). (Also check
out theJeroen Ooms's [hunspell](https://github.com/ropensci/hunspell)
package, which is a part of rOpensci.)

One of the basic tasks in any NLP pipeline is turning texts (which
humans can read) into tokens (which machines can compute with). For
example, you might break a text into words or into
[n-grams](https://en.wikipedia.org/wiki/N-gram). Here is an example
using the [former slave
interviews](https://memory.loc.gov/ammem/snhtml/snhome.html) from the
Great Depression era Federal Writers' Project. (A data package with
those interviews is in development
[here](https://github.com/lmullen/WPAnarratives)).

    # devtools::install_github("lmullen/WPAnarratives")# install.packages("tokenizers")library(WPAnarratives)library(tokenizers)text <- head(wpa_narratives$text, 5)class(text)## [1] "character"words <- tokenize_words(text, lowercase = TRUE)str(words)## List of 5##  $ : chr [1:1141] "_he" "loved" "young" "marster" ...##  $ : chr [1:1034] "_old" "joe" "can" "keep" ...##  $ : chr [1:824] "_jesus" "has" "my" "chillun" ...##  $ : chr [1:779] "charity" "anderson" "who" "believes" ...##  $ : chr [1:350] "dat" "was" "one" "time" ...ngrams <- tokenize_ngrams(text, n_min = 3, n = 5)str(ngrams)## List of 5##  $ : chr [1:3414] "_he loved young" "_he loved young marster" "_he loved young marster john_" "loved young marster" ...##  $ : chr [1:3093] "_old joe can" "_old joe can keep" "_old joe can keep his" "joe can keep" ...##  $ : chr [1:2463] "_jesus has my" "_jesus has my chillun" "_jesus has my chillun counted_" "has my chillun" ...##  $ : chr [1:2328] "charity anderson who" "charity anderson who believes" "charity anderson who believes she" "anderson who believes" ...##  $ : chr [1:1041] "dat was one" "dat was one time" "dat was one time when" "was one time" ...

Practically all text analysis packages provide their own functions for
tokenizing text, so why do R users need this package?

First, these tokenizers are reasonably fast. The basic string operations
are handled by the
[stringi](https://cran.r-project.org/package=stringi/) package, which is
quick while also doing the correct thing across encodings and locales.
And [Dmitriy Selivanov](https://dsnotes.com/) (author of the text2vec
package) has written the n-gram and skip n-gram tokenizers in C++ so
that those are fast too. It is probably possible to write tokenizers
with better performance, but these are fast enough for even large scale
text mining efforts.

The second and more important reason is that these tokenizers are
consistent. They all take either a character vector of any length, or a
list where each element is a character vector of length one. The idea is
that each element of the input comprises a text. Then each function
returns a list with the same length as the input vector, where each
element in the list contains the tokens generated by the function. If
the input character vector or list is named, then the names are
preserved, so that the names can serve as identifiers.

And third, the tokenizers are reasonably comprehensive, including
functions for characters, lines, words, word stems, sentences,
paragraphs, n-grams, skip n-grams, and regular expressions.

My hope is that developers of other text analysis packages for R will
rely on this package to provide tokenizers. (So far only tidytext has
taken me up on that, but I also have to re-write my own
[textreuse](https://github.com/ropensci/textreuse) package now.) But
even if natural language packages do not take the package as a formal
dependency, most packages let you pass in your own tokenizing functions.
So users can reap the benefits of a consistent set of tokenizers by
using the functions in this package. The success of the
"[tidyverse](https://twitter.com/hadleywickham/status/751805589425000450)"
has shown the power of buying into a convention for the structure of
data and the inputs and outputs of functions. My hope is that the
tokenizers package is a step in that direction for text analysis in R.


The R package ecosystem for natural language processing has been flourishing in recent days. R packages for text analysis have usually been based on the classes provided by the NLP or tm packages. Many of them depend on Java. But recently there have been a number of new packages for text analysis in R, most notably text2vec, quanteda, and tidytext.

Messaggi di Rogue Scholar

Ozymandias meets Wikipedia, with notes on natural language generation

New package tokenizers joins rOpenSci