University of Glasgow

Roderic Page

I\'ve released a simple search engine for publications in Wikidata.
[Wikicite Search](https://wikicite-search.herokuapp.com) takes its name
from the [WikiCite](http://wikicite.org) project, which was an
initiative to create a bibliographic database in Wikidata. Since
bibliographic data is a core component of taxonomic research (arguably
taxonomy is mostly tracing the fate of the \"tags\" we call taxonomic
names) I\'ve spent some time getting taxonomic literature into Wikidata.
Since there are bots already adding articles by harvesting sources such
as CrossRef and PubMed, I\'ve focussed on literature that is harder to
add, such as articles with non-CrossRef DOIs, or those without DOIs at
all.

Once you have a big database, you are then faced with the challenge of
finding things in that database. Wikidata supports generic search, but I
wanted something more closely geared to bibliographic data. Hence
[Wikicite Search](https://wikicite-search.herokuapp.com). Over the last
few years I\'ve made several attempts at a bibliographic search engine,
for this project I\'ve finally settled on some basic ideas:

1.  The core data structure is CSL-JSON, a simple but rich JSON format
    for expressing bibliographic data.
2.  The search engine is Elasticsearch. The documents I upload include
    the CSL-JSON for an article, but also a simple text representation
    of the article. This text representation may include multiple
    languages if, for example, the article has a title in more than one
    language. This means that if an article has both English and Chinese
    titles you can find it searching in either language.
3.  The web interface is very simple: search for a term, get results. If
    the search term is a Wikidata identifier you get just the
    corresponding article, e.g.
    [Q98715368](https://wikicite-search.herokuapp.com/?q=Q98715368).
4.  There is a [reconciliation
    API](https://wikicite-search.herokuapp.com/match.html) to help match
    articles to the database. Paste in one citation per line and you get
    back matches (if found) for each citation.
5.  Where possible I display a link to a PDF of the article, which is
    typically stored in the Internet Archive or accessible via the
    Wayback Machine.

There are millions of publications in Wikidata, currently less than half
a million are in my search engine. My focus is narrowly on eukaryote
taxonomy and related topics. I will be adding more articles as time
permits. I also periodically reload existing articles to capture updates
to the metadata made by the Wikidata community - being a wiki the data
in Wikidata is constantly evolving.

My goal is to have a simple search tool that focusses on matching
citation strings. In other words, it is designed to find a reference you
are looking for, rather than be a tool to search the taxonomic
literature. If that sounds contradictory, consider that my tool will
only find a paper about a taxon if it is explicitly named in the title.
A more sophisticated search engine would support things like synonym
resolution, etc.

The other reason I built this is to provide an API for accessing
Wikidata items and displaying them in other formats. For example, an
article in the WikiCite search engine can be retrieved in CSL-JSON
format, or in RDF as JSON-LD.

As always, it\'s very early days. But I don\'t think it\'s unreasonable
to imagine that as Wikidata grows we could envisage having a search
engine that includes the bulk of the taxonomic literature.


I've released a simple search engine for publications in Wikidata. Wikicite Search takes its name from the WikiCite project, which was an initiative to create a bibliographic database in Wikidata. Since bibliographic data is a core component of taxonomic research (arguably taxonomy is mostly tracing the fate of the "tags" we call taxonomic names) I've spent some time getting taxonomic literature into Wikidata.

Towards a WikiCite search engine

Scott Chamberlain

## DBI

What is DBI? [DBI](https://cran.r-project.org/web/packages/DBI/) is an R
package. It defines an interface to relational database management
systems (R/DBMS) that other R packages build upon to interact with a
specific relational database, such as [SQLite](https://sqlite.org/) or
[PostgreSQL](https://www.postgresql.org/).

## NoSQL

[NoSQL databases](https://nosql-database.org/) are a very broad class of
database that can include document databases such as CouchDB and
MongoDB, key-value stores such as Redis, and more. They are generally
not row-column relational stores though, though can include that. NoSQL
is often thought of now as "not only SQL".

You can imagine how it is relatively straight-forward to create a common
interace to row-column oriented databases, and DBI is great for that.

However, a common interface to NoSQL datbases is a bit harder to wrap
your head around for various reasons. One of the most obvious of which
is that they don't share a fundamental data structure like R/DMBS do.

## A sort of DBI for NoSQL

[nodbi](https://github.com/ropensci/nodbi) (the `no` for "N" and "O" in
"NoSQL") was started nearly 3 years ago during the [2015 rOpenSci
Unconference](https://github.com/ropensci/unconf15) on [26
March](https://github.com/ropensci/nodbi/commits/29340320b28f13e942ae177886d3d6b87366c4c2)
together with [Rich FitzJohn](https://github.com/richfitz) and [Jeroen
Ooms](https://github.com/jeroen). We've worked on it here and there, but
it hasn't seen any commits in nearly a year.

Given the success of
[dplyr](https://cran.r-project.org/web/packages/dplyr/) it goes to
reason that something that can abstract away the details for NoSQL
databases - for a subset of the most common operations - could prove to
be very useful.

nodbi takes a different approach than DBI (thus the "sort of" in this
section title) in that instead of package X for database Y importing DBI
and extending its methods, nodbi provides the connection objects to
various databases, and a set of common methods for interacting with
those databases. Whereas DBI doesn't import or depend in any way on
packages for individual databases, nodbi has packages for individual
databases in Suggests. That is to say, nodbi can be installed and loaded
without any specific database package, but to use any one database
(e.g., Mongo) you need to install the package for that database (i.e.,
[mongolite](https://cran.r-project.org/web/packages/mongolite/)).
Because database packages in nodbi are in Suggests, you can pick and
choose the packages you want to have installed - and forget about the
others.

Check out the package [on GitHub](https://github.com/ropensci/nodbi).

## data.frame's

Thus far, we've built nodbi around data.frame's. That is, we're focusing
on the data.frame use case as it's very common that R users are dealing
solely with data.frame's in their analysis pipelines.

Let us know if you have a strong need for lists or vectors, etc. in
nodbi.

## supported databases

Connection functions to each database follow the pattern `src_`
following the pattern used by `dplyr`

- MongoDB: `src_mongo`
- Redis: `src_redis`
- CouchDB: `src_couchdb`
- Elasticsvearch: `src_elastic`
- Etcd: `src_etcd`

The details of connecting to different databases vary as you'd imagine,
but what's returned from `src_` functions can be passed to any of the
below methods (for the most part).

## supported methods

- create (`docdb_create`): Create an object
- get (`docdb_get`): Get an object
- delete (`docdb_delete`): Delete an object
- update (`docdb_update`): Update an object. CouchDB only for now

Operations in nodbi follow a key-value pattern. That is, an object
(read: data.frame) is stored in your choice of database with a key
(read: character string). Not every operation above needs both the key
and value - for example, deleting an object only requires the key.

Internally, a key maps differently to Redis vs. Mongo vs. CouchDB, etc.
But the point is that we're abstracting away the details so that the
user can focus on getting things done faster.

## Setup

You'll need to start up the database you want to work with if it's not
started already. Let's work with Mongo for this blog post.

Check out the [mongolite book](https://jeroen.github.io/mongolite/) for
details on installing MongoDB

Start up mongo on your cli (i.e., shell or terminal)

``` {tabindex="0"}
mongod # or sudo mongod
```

Install database packages

::: {.highlight}
``` {.chroma tabindex="0"}
install.packages(c("sofa", "etseed", "elastic", "mongolite", "redux"))
```
:::

Install nodbi

::: {.highlight}
``` {.chroma tabindex="0"}
devtools::install_github("ropensci/nodbi")
```
:::

::: {.highlight}
``` {.chroma tabindex="0"}
library(nodbi)
```
:::

## Examples

Connect to Mongo with `src_mongo`

::: {.highlight}
``` {.chroma tabindex="0"}
(src <- src_mongo(verbose = FALSE))#> MongoDB 3.6.2 (uptime: 3017s)#> URL: leothelion.local/test
```
:::

With the connection object you can access the name of the Mongo database

::: {.highlight}
``` {.chroma tabindex="0"}
src$db#> [1] "test"
```
:::

And the connection `mongolite` connection object

::: {.highlight}
``` {.chroma tabindex="0"}
src$con#> <Mongo collection> 'test' #>  $aggregate(pipeline = "{}", options = "{\"allowDiskUse\":true}", handler = NULL, pagesize = 1000) #>  $count(query = "{}") #>  $distinct(key, query = "{}") #>  $drop() #>  $export(con = stdout(), bson = FALSE) #>  $find(query = "{}", fields = "{\"_id\":0}", sort = "{}", skip = 0, limit = 0, handler = NULL, pagesize = 1000) #>  $import(con, bson = FALSE) #>  $index(add = NULL, remove = NULL) #>  $info() #>  $insert(data, pagesize = 1000, stop_on_error = TRUE, ...) #>  $iterate(query = "{}", fields = "{\"_id\":0}", sort = "{}", skip = 0, limit = 0) #>  $mapreduce(map, reduce, query = "{}", sort = "{}", limit = 0, out = NULL, scope = NULL) #>  $remove(query, just_one = FALSE) #>  $rename(name, db = NULL) #>  $replace(query, update = "{}", upsert = FALSE) #>  $update(query, update = "{\"$set\":{}}", filters = NULL, upsert = FALSE, multiple = FALSE)
```
:::

Insert a data.frame (the returned object is from `mongolite`) using
`docdb_create`

::: {.highlight}
``` {.chroma tabindex="0"}
library("ggplot2")x <- docdb_create(src = src, key = "diamonds", value = diamonds)x#> List of 5#>  $ nInserted  : num 53940#>  $ nMatched   : num 0#>  $ nRemoved   : num 0#>  $ nUpserted  : num 0#>  $ writeErrors: list()
```
:::

Get the data.frame back from Mongo using `docdb_get`

::: {.highlight}
``` {.chroma tabindex="0"}
head(docdb_get(src, "diamonds"))#>   carat       cut color clarity depth table price    x    y    z#> 1  0.23     Ideal     E     SI2  61.5    55   326 3.95 3.98 2.43#> 2  0.21   Premium     E     SI1  59.8    61   326 3.89 3.84 2.31#> 3  0.23      Good     E     VS1  56.9    65   327 4.05 4.07 2.31#> 4  0.29   Premium     I     VS2  62.4    58   334 4.20 4.23 2.63#> 5  0.31      Good     J     SI2  63.3    58   335 4.34 4.35 2.75#> 6  0.24 Very Good     J    VVS2  62.8    57   336 3.94 3.96 2.48
```
:::

Delete the object using `docdb_delete`

::: {.highlight}
``` {.chroma tabindex="0"}
docdb_delete(src, "diamonds")
```
:::

The same pattern as above can be used for any of the other database
options, except the connecting with `src_*` differs among databases.

## similar art

The one that comes closest is [storr](https://github.com/richfitz/storr)
by [Rich FitzJohn](https://github.com/richfitz) - which shares Redis in
common with nodbi as a backend. storr seems to be more focused on the
caching use case - whereas, nodbi focuses on user X has a specific
database they keep their data in and they'd like to not worry about the
details of working with that database. Perhaps the distinction isn't
really there, I'm not sure.

## Todo

- We plan to add support for more databases, let us know what you want
  supported
- We may add more operations in addition to create/get/delete/update.
  What things do you want nodbi to do?
- [Fetching partial
  data.frame's](https://github.com/ropensci/nodbi/issues/10)
- Speed ups of course, in due time
- Eventually we'll push a version to CRAN - we're hoping to get more
  users kicking the tires before that.

## Feedback!

We'd love to get some eyes on this; to sort out problems that will no
doubt arise from real world scenarios; to flesh out new use cases we
hadn't thought of, etc.

Open an issue: <https://github.com/ropensci/nodbi/issues/new>.


DBI   What is DBI? DBI is an R package. It defines an interface to relational database management systems (R/DBMS) that other R packages build upon to interact with a specific relational database, such as SQLite or PostgreSQL. NoSQL   NoSQL databases are a very broad class of database that can include document databases such as CouchDB and MongoDB, key-value stores such as Redis, and more.

nodbi: the NoSQL Database Connector

**elastic** is an R client for
[Elasticsearch](https://www.elastic.co/products/elasticsearch)

`elastic` has been around since 2013, with the first commit in
[November,
2013](https://github.com/ropensci/elastic/commit/f7b04589b2cb711a21223bb4f20b34bc9330ef8d).

> sidebar - 'elastic' was picked as a package named before the company
> now known as Elasticchanged their name to Elastic.

## What is Elasticsearch?

If you aren't familiar with Elasticsearch, it is a distributed, RESTful
search and analytics engine.It's similar to
[Solr](https://lucene.apache.org/solr//). It falls in the NoSQL bin of
databases, holding data in JSON documents, insteadof rows and columns.
Elasticsearch has a concept of **index**, similar to a database in
SQL-land.You can hold many documents of similar type within a single
index. There is powerful searchcapabilities, including lots of different
types of queries that can be done separatelyor combined. And best of all
it's super fast.

## Other clients

The Elastic company maintains some official clients, including the
Python
client[elasticsearch-py](https://elasticsearch-py.readthedocs.io/en/master/),
and it's higherlevel DSL client
[elasticsearch-dsl](https://elasticsearch-dsl.readthedocs.io/en/latest/).

I won't talk much about it, but we have slowly been working on an R
equivalent of thePython DSL client, called
[elasticdsl](https://github.com/ropensci/elasticdsl), fora human
friendly way to compose Elasticsearch queries.

## Vignettes

Check out the [elastic introduction
vignette](https://cran.rstudio.com/web/packages/elastic/vignettes/elastic.html)and
the [search
vignette](https://cran.rstudio.com/web/packages/elastic/vignettes/search.html)
to get started.

## Noteable features

- `elastic` has nearly complete coverage of the Elasticsearch HTTP API.
  If there'sanything missing you need in this client, let us know! Check
  out the[features
  label](https://github.com/ropensci/elastic/issues?q=is%3Aissue+is%3Aopen+label%3Afeatures)for
  features we plan to add to the package.
- We fail well. This is important to us. We allow the user to choose
  simple errorsto just give e.g., 404 HTTP error, or complex errors,
  including full stack tracefrom Elasticsearch in addition to the HTTP
  errror. We strive to fail well whenusers give the wrong type of input,
  etc. as well. Let us know if `elastic` is notfailing well!
- We strive to allow R centric ways of interacting with Elasticsearch.
  For example,in the function `docs_bulk`, our interface to the
  Elasticsearch [bulk
  API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html)we
  make it easy to create documents in your Elasticsearch instance from R
  lists,data.frame's and from bulk format files on disk.
- `elastic` works with most versions of Elasticsearch. We run the test
  suite on 11versions of Elasticsearch, from `v1.0.0` up to `v5.5.0`. We
  strive to fail wellwith useful messages when there is a feature no
  longer available or one that isa new feature and not available in
  previous Elasticsearch versions.
- Search inputs are flexible: lists and JSON strings both work.
- Arguably, a noteable feature is that this client has been around
  nearly 4 years,so we've surfaced and squashed many bugs.

## Getting help

- If you have a bug or a feature request, post it in the repo at
  <https://github.com/ropensci/elastic/issues>
- Stackoverflow: Check out combination of the tags `[elasticsearch]` and
  `[r]` <https://stackoverflow.com/questions/tagged/elasticsearch+r>
- Reach out to me on Twitter at
  [@sckottie](https://twitter.com/sckottie)
- Post question/problem in the [rOpenSci discussion
  forum](https://discuss.ropensci.org/)
- Email me [directly](mailto:myrmecocystus@gmail.com)

## Setup

Install `elastic`

::: {.highlight}
``` {.chroma tabindex="0"}
install.packages("elastic")
```
:::

Or get the development version:

::: {.highlight}
``` {.chroma tabindex="0"}
devtools::install_github("ropensci/elastic")
```
:::

::: {.highlight}
``` {.chroma tabindex="0"}
library(elastic)
```
:::

I'm running Elasticsearch version:

::: {.highlight}
``` {.chroma tabindex="0"}
ping()$version$number#> [1] "5.4.0"
```
:::

## Examples

### Initialize a client

Using `connect()`

::: {.highlight}
``` {.chroma tabindex="0"}
elastic::connect()#> transport:  http#> host:       127.0.0.1#> port:       9200#> path:       NULL#> username:   NULL#> password:   <secret>#> errors:     simple#> headers (names):  NULL
```
:::

By default, you connect to `localhost` and port `9200`. There's
paramatersfor setting transport schema, username, password, and base
search path (e.g.,`_search` or something else).

> See bottom of post about possible changes in connections.

### Get some data

Elasticsearch has a bulk load API to load data in fast. The format is
pretty weirdthough. It's sort of JSON, but would pass no JSON linter. I
include a few datasets in `elastic` so it's easy to get up and running,
and so when you run examplesin this package they'll actually run the
same way (hopefully).

### Public Library of Science (PLOS) data

A dataset inluded in the `elastic` package is metadata for PLOS
scholarly articles.Get the file path, then load:

::: {.highlight}
``` {.chroma tabindex="0"}
plosdat <- system.file("examples", "plos_data.json", package = "elastic")invisible(docs_bulk(plosdat))
```
:::

### Search

The main search function is `Search()`. Running it without any inputs
searchesacross all indices - in this case only the `plos` index.

::: {.highlight}
``` {.chroma tabindex="0"}
Search()
```
:::

``` {tabindex="0"}
#> $took#> [1] 1#>#> $timed_out#> [1] FALSE#>#> $`_shards`#> $`_shards`$total#> [1] 5#>#> $`_shards`$successful#> [1] 5#>#> $`_shards`$failed#> [1] 0...
```

Search just the `plos` index and only return 1 result

::: {.highlight}
``` {.chroma tabindex="0"}
Search(index = "plos", size = 1)$hits$hits#> [[1]]#> [[1]]$`_index`#> [1] "plos"#>#> [[1]]$`_type`#> [1] "article"#>#> [[1]]$`_id`#> [1] "0"#>#> [[1]]$`_score`#> [1] 1#>#> [[1]]$`_source`#> [[1]]$`_source`$id#> [1] "10.1371/journal.pone.0007737"#>#> [[1]]$`_source`$title#> [1] "Phospholipase C-β4 Is Essential for the Progression of the Normal Sleep Sequence and Ultradian Body Temperature Rhythms in Mice"
```
:::

Search the `plos` index, and the `article` document type, sort by title,
and query for *antibody*, limit to 1 result.

First, with Elasticsearch `v5` and greater, we need to set
`fielddata = true` if we want to search on or sort on a text field.

::: {.highlight}
``` {.chroma tabindex="0"}
mapping_create("plos", "article", update_all_types = TRUE, body = '{   "properties": {     "title": {     "type":     "text",     "fielddata": true   } }}')#> $acknowledged#> [1] TRUESearch(index = "plos", type = "article", sort = "title", q = "antibody", size = 1)$hits$hits#> [[1]]#> [[1]]$`_index`#> [1] "plos"#>#> [[1]]$`_type`#> [1] "article"#>#> [[1]]$`_id`#> [1] "568"#>#> [[1]]$`_score`#> NULL#>#> [[1]]$`_source`#> [[1]]$`_source`$id#> [1] "10.1371/journal.pone.0085002"#>#> [[1]]$`_source`$title#> [1] "Evaluation of 131I-Anti-Angiotensin II Type 1 Receptor Monoclonal Antibody as a Reporter for Hepatocellular Carcinoma"#>#>#> [[1]]$sort#> [[1]]$sort[[1]]#> [1] "1"
```
:::

### Get documents

Get document with `id=1`

::: {.highlight}
``` {.chroma tabindex="0"}
docs_get(index = 'plos', type = 'article', id = 1)#> $`_index`#> [1] "plos"#>#> $`_type`#> [1] "article"#>#> $`_id`#> [1] "1"#>#> $`_version`#> [1] 1#>#> $found#> [1] TRUE#>#> $`_source`#> $`_source`$id#> [1] "10.1371/journal.pone.0098602"#>#> $`_source`$title#> [1] "Population Genetic Structure of a Sandstone Specialist and a Generalist Heath Species at Two Levels of Sandstone Patchiness across the Strait of Gibraltar"
```
:::

Get certain fields

::: {.highlight}
``` {.chroma tabindex="0"}
docs_get(index = 'plos', type = 'article', id = 1, fields = 'id')#> $`_index`#> [1] "plos"#>#> $`_type`#> [1] "article"#>#> $`_id`#> [1] "1"#>#> $`_version`#> [1] 1#>#> $found#> [1] TRUE
```
:::

### Raw JSON data

You can optionally get back raw JSON from many functions by setting
parameter `raw=TRUE`.

For example, get raw JSON, then parse with `jsonlite`

::: {.highlight}
``` {.chroma tabindex="0"}
(out <- docs_mget(index = "plos", type = "article", id = 5:6, raw = TRUE))#> [1] "{\"docs\":[{\"_index\":\"plos\",\"_type\":\"article\",\"_id\":\"5\",\"_version\":1,\"found\":true,\"_source\":{\"id\":\"10.1371/journal.pone.0085123\",\"title\":\"MiR-21 Is under Control of STAT5 but Is Dispensable for Mammary Development and Lactation\"}},{\"_index\":\"plos\",\"_type\":\"article\",\"_id\":\"6\",\"_version\":1,\"found\":true,\"_source\":{\"id\":\"10.1371/journal.pone.0098600\",\"title\":\"Correction: Designing Mixed Species Tree Plantations for the Tropics: Balancing Ecological Attributes of Species with Landholder Preferences in the Philippines\"}}]}"#> attr(,"class")#> [1] "elastic_mget"jsonlite::fromJSON(out)#> $docs#>   _index   _type _id _version found                   _source.id#> 1   plos article   5        1  TRUE 10.1371/journal.pone.0085123#> 2   plos article   6        1  TRUE 10.1371/journal.pone.0098600#>                                                                                                                                                     _source.title#> 1                                                                       MiR-21 Is under Control of STAT5 but Is Dispensable for Mammary Development and Lactation#> 2 Correction: Designing Mixed Species Tree Plantations for the Tropics: Balancing Ecological Attributes of Species with Landholder Preferences in the Philippines
```
:::

### Aggregation search

Here, we'll use another dataset that comes with the package on
Shakespeare plays.

::: {.highlight}
``` {.chroma tabindex="0"}
gbifdat <- system.file("examples", "gbif_data.json", package = "elastic")invisible(docs_bulk(gbifdat))
```
:::

Define an aggregation query:

::: {.highlight}
``` {.chroma tabindex="0"}
aggs <- '{    "aggs": {        "latbuckets" : {           "histogram" : {               "field" : "decimalLatitude",               "interval" : 5           }        }    }}'
```
:::

Search the `gbif` index

::: {.highlight}
``` {.chroma tabindex="0"}
res <- Search(index = "gbif", body = aggs, size = 0)$aggregations$latbuckets$bucketsdo.call("rbind.data.frame", res)#>    key doc_count#> 2  -35         1#> 22 -30         0#> 3  -25         0#> 4  -20         0#> 5  -15         0#> 6  -10         0#> 7   -5         1#> 8    0         0#> 9    5         0#> 10  10         0#> 11  15         0#> 12  20         0#> 13  25         4#> 14  30         2#> 15  35         3#> 16  40         2#> 17  45        66#> 18  50       183#> 19  55       487#> 20  60       130#> 21  65        20
```
:::

### Scrolling search - instead of paging

When you want all the documents, your best bet is likely to be
[scrolling
search](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html).

Here's an example. First, use `Search()`, setting a value for the
`scroll` parameter.

::: {.highlight}
``` {.chroma tabindex="0"}
res1 <- Search(index = 'shakespeare', scroll = "1m")
```
:::

You get a scroll ID back when setting the `scroll` parameter

::: {.highlight}
``` {.chroma tabindex="0"}
res1$`_scroll_id`#> [1] "DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAAElFnZ2X3FJVWEyUU1HQjl2cFpWUFl0cXcAAAAAAAABJBZ2dl9xSVVhMlFNR0I5dnBaVlBZdHF3AAAAAAAAAScWdnZfcUlVYTJRTUdCOXZwWlZQWXRxdwAAAAAAAAEmFnZ2X3FJVWEyUU1HQjl2cFpWUFl0cXcAAAAAAAABIxZ2dl9xSVVhMlFNR0I5dnBaVlBZdHF3"
```
:::

Use a while loop to get all results

::: {.highlight}
``` {.chroma tabindex="0"}
out1 <- list()hits <- 1while (hits != 0) {  tmp1 <- scroll(scroll_id = res1$`_scroll_id`)  hits <- length(tmp1$hits$hits)  if (hits > 0) {   out1 <- c(out1, tmp1$hits$hits)  }}
```
:::

Woohoo! Collected all 1 documents in very little time.

Now, get `_source` from each document:

::: {.highlight}
``` {.chroma tabindex="0"}
docs <- lapply(out1, "[[", "_source")length(docs)#> [1] 4988vapply(docs[1:10], "[[", "", "text_entry")#>  [1] "Without much shame retold or spoken of."#>  [2] "For more uneven and unwelcome news"#>  [3] "And shape of likelihood, the news was told;"#>  [4] "Mordake the Earl of Fife, and eldest son"#>  [5] "It is a conquest for a prince to boast of."#>  [6] "Amongst a grove, the very straightest plant;"#>  [7] "That some night-tripping fairy had exchanged"#>  [8] "Then would I have his Harry, and he mine."#>  [9] "This is his uncles teaching; this is Worcester,"#> [10] "Malevolent to you in all aspects;"
```
:::

### Bulk documents

You've already seen the bulk docs API in action above. Above though, we
wereusing `docs_bulk.character` - where the input is a character string
that's afile path.

Here, I'll describe briefly how you can insert any data.frame as
documents in yourElasticsearch instance. We'll use the diamonds dataset
from the \~54K row `ggplot2`package.

``` {tabindex="0"}
#> $acknowledged#> [1] TRUE
```

::: {.highlight}
``` {.chroma tabindex="0"}
library(ggplot2)invisible(docs_bulk(diamonds, "diam"))#> |==================================| 100%
```
:::

::: {.highlight}
``` {.chroma tabindex="0"}
Search("diam")$hits$total#> [1] 47375
```
:::

That's pretty easy! This function is used a lot, particularly with
data.frame's - sowe get many questions/feedback on this so it will just
keep getting better/faster.

## TO DO

### Connections

We're planning to roll out changes in how you connect to Elasticsearch
from `elastic`.Right now, you can only connect to one Elasticsearch
instance per R session -your details are set and then recalled
internally in each function. We plan to changethis to instantiate a
client and then you either call functions on the client(e.g., using
`R6`) or pass the client object onto functions.

Checkout [issue \#87](https://github.com/ropensci/elastic/issues/87) to
followprogress or discuss.

### Move to using crul for http

`crul` is a relatively new R http client - and has async baked in - as
well as mocking.Development should be easier with it as I can mock
requests for test suites, andallow users to toggle async more easily.

## Call to action

We can use your help! Elasticsearch development moves pretty fast - we'd
love this client towork with every single Elasticsearch version to the
extent possible - and we'd love tosquash every bug and solve every
feature request fast.

If you need to use Elasticsearch from R, please try out `elastic`!

- Report bugs!
- File feature requests!
- Send PR's!


<strong>
 elastic
</strong>
is an R client for Elasticsearch  elastic has been around since 2013, with the first commit in November, 2013. What is Elasticsearch? If you aren’t familiar with Elasticsearch, it is a distributed, RESTful search and analytics engine.It’s similar to Solr. It falls in the NoSQL bin of databases, holding data in JSON documents, insteadof rows and columns.

elastic - Elasticsearch for R

Following on from earlier posts [exploring how to map DNA
barcodes](http://iphylo.blogspot.com/2016/05/notes-on-next-steps-for-million-dna.html)
and [putting barcodes into
GBIF](http://iphylo.blogspot.co.uk/2016/12/ibol-dna-barcodes-in-gbif.html)
it\'s time to think about taking advantage of what makes barcodes
different from typical occurrence data. At present GBIF displays data as
dots on a map (as do I in <http://iphylo.org/~rpage/bold-map/>). But
barcodes come with a lot more information than that. I\'m interested in
exploring how we might measure and visualise biodiversity using just
sequences.

Based on a talk by Zachary Tong ([Going Organic - Genomic sequencing in
Elasticsearch](https://speakerdeck.com/polyfractal/going-organic-genomic-sequencing-in-elasticsearch))
I\'ve started to play with *n*-gram searches on DNA barcodes using
[Elasticsearch](https://www.elastic.co/products/elasticsearch), an open
source search engine. The idea is that we break the DNA sequence into
every possible \"word\" of length *n* (also called a *k*-mer or
*k*-tuple, where *k* = *n*).

For example, for *n* = 5, the sequence GTATCGGTAACGAACTT would look like
this:

    GTATCGGTAACGAACTT

    GTATC
     TATCG
      ATCGG
       TCGGT
        CGGTA
         GGTAA
          GTAAC
           TAACG
            AACGAA
             ACGAAC
              CGAACT
               GAACTT

The sequence GTATCGGTAACGAACTT comes from Hajibabaei and Singer (2009)
who discussed \"Googling\" DNA sequences using search engines (see also
Kuksa and Pavlovic, 2009). If we index sequences in this way then we can
do BLAST-like searches very quickly using Elasticsearch. This means
it\'s feasible to take a DNA barcode and ask \"what sequences look like
this?\" and return an answer qucikly enoigh for a user not to get bored
waiting.

Another nice feature of Elasticsearch is that it supports geospatial
queries, so we can ask for, say, all the barcodes in a particular
region. Having got such a list, what we really want is not a list of
sequences but a phylogenetic tree. Traditionally this can be a time
consuming operation, we have to take the sequences, align them, then
input that alignment into a tree building algorithm. Or do we?

There\'s growing interest in \"alignment-free\" phylogenetics, a phrase
I\'d heard but not really followed up. Yang and Zhang (2008) described
an approach where every sequences is encoded as a vector of all possible
*k*-tuples. For DNA sequences *k* = 5 there are 4^5^ = 1024 possible
combinations of the bases A, C, G, and T, so a sequence is represented
as a vector with 1024 elements, each one is the frequency of the
corresponding 5-tuple. The \"distance\" between two sequences is the
mathematical distance between these vectors for the two sequences. Hence
we no longer need to align the sequences being comapred, we simply chunk
them into all \"words\" of 5 bases in length, and compare the
frequencies of the 1024 different possible \"words\".

In their study Yang and Zhang (2008) found that:

> We compared tuples of diﬀerent sizes and found that tuple size 5
> combines both performance speed and accuracy; tuples of shorter
> lengths contain less information and include more randomness; tuples
> of longer lengths contain more information and less random- ness, but
> the vector size expands exponentially and gets too large and
> computationally ineﬃcient.

So we can use the same word size for both Elasticsearch indexing and for
computing the distance matrix. We still need to create a tree, for which
we could use something quick like neighbour-joining (NJ). This method is
sufficiently quick to be available in Javascript and hence can be
computed by a web browser (e.g.,
[biosustain/neighbor-joining](https://github.com/biosustain/neighbor-joining)).

Putting this all together, I\'ve built a rough-and-ready demo that takes
some DNA barcodes, puts them on a map, then enables you to draw a box on
a map and the demo will retrieve the DNA barcodes in that area, compute
a distance matrix using 5-tuples, then build a NJ tree, all on the fly
in your web browser.

::::: {style="text-align:center"}
:::: {.iframe}
[](https://player.vimeo.com/video/209886913)

::: {#crawler_player}
Play

![Vimeo](https://f.vimeocdn.com/p/images/crawler_logo.png){.logo}
:::
::::

[Phylodiversity on the fly](https://vimeo.com/209886913) from [Roderic
Page](https://vimeo.com/rdmpage) on [Vimeo](https://vimeo.com).
:::::

This is all very crude, and I need to explore scalability (at the moment
I limit the results to the first 200 DNA sequences found), but it\'s
encouraging. I like the idea that, in principle, we could go to any part
of the globe, ask \"what\'s there?\" and get back a phylogenetic tree
for the DNA barcodes in that area.

This also means that we could start exploring phylogenetic diversity
using DNA barcodes, as Faith & Baker (2006) wanted a decade ago:

> \...PD has been advocated as a way to make the best-possible use of
> the wealth of new data expected from large-scale DNA "barcoding"
> programs. This prospect raises interesting bio-informatics issues
> (discussed below), including how to link multiple sources of evidence
> for phylogenetic inference, and how to create a web-based linking of
> PD assessments to the barcode--of-life database (BoLD).

The phylogenetic diversity of an area is essentially the length of the
tree of DNA barcodes, so if we build a tree we have a measure of
diversity. Note that this contrasts with other approaches, such as
Miraldo et al.\'s \"An Anthropocene map of genetic diversity\" which
measured genetic diversity within species but not between (!).

#### Practical issues

There are a bunch of practical issues to work through, such as how
scalable it is to compute phylogenies using Javascript on the fly. For
example, could we do something like generate a one degree by one degree
grid of the Earth, take all the barcodes in each cell and compute a
phylogeny for each cell? Could we do this in CouchDB? What about
sampling, should we be taking a finite, random sample of sequences so
that we try and avoid sampling bias?

There are also data management issues. I\'m exploring downloading DNA
barcodes, creating a Darwin Core Archive file using the [Global Genome
Biodiversity Network (GGBN) data
standard](http://wiki.ggbn.org/ggbn/Main_Page), then converting the
Darwin Core Archive into JSON and sending that to Elasticsearch. The
reason for the intermediate step of creating the archive is so that we
can edit the data, add missing geospatial informations, etc. I envisage
having a set of archives, hosted say on GitHub. These archives could
also be directly imported into GBIF, ready for the time that GBIF can
handle genomic data.

#### References

- Faith, D. P., & Baker, A. M. (2006). Phylogenetic diversity (PD) and
  biodiversity conservation: some bioinformatics challenges. Evol
  Bioinform Online. 2006; 2: 121--128.
  [PMC2674678](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2674678/)

  ::: {.altmetric-embed badge-type="donut" pmid="19455206"}
  :::
- Hajibabaei, M., & Singer, G. A. (2009). Googling DNA sequences on the
  World Wide Web. BMC Bioinformatics. Springer Nature.
  <https://doi.org/10.1186/1471-2105-10-s14-s4>

  ::: {.altmetric-embed badge-type="donut" doi="10.1186/1471-2105-10-s14-s4"}
  :::
- Kuksa, P., & Pavlovic, V. (2009). Efficient alignment-free DNA barcode
  analytics. BMC Bioinformatics. Springer Nature.
  <https://doi.org/10.1186/1471-2105-10-s14-s9>

  ::: {.altmetric-embed badge-type="donut" doi="10.1186/1471-2105-10-s14-s9"}
  :::
- Miraldo, A., Li, S., Borregaard, M. K., Florez-Rodriguez, A.,
  Gopalakrishnan, S., Rizvanovic, M., ... Nogues-Bravo, D. (2016,
  September 29). An Anthropocene map of genetic diversity. Science.
  American Association for the Advancement of Science (AAAS).
  <https://doi.org/10.1126/science.aaf4381>

  ::: {.altmetric-embed badge-type="donut" doi="10.1126/science.aaf4381"}
  :::
- Yang, K., & Zhang, L. (2008, January 10). Performance comparison
  between k-tuple distance and four model-based distances in
  phylogenetic tree reconstruction. Nucleic Acids Research. Oxford
  University Press (OUP). <https://doi.org/10.1093/nar/gkn075>

  ::: {.altmetric-embed badge-type="donut" doi="10.1093/nar/gkn075"}
  :::


Efficient alignment-free DNA barcode analytics

Performance comparison between k-tuple distance and four model-based distances in phylogenetic tree reconstruction

Following on from earlier posts exploring how to map DNA barcodes and putting barcodes into GBIF it's time to think about taking advantage of what makes barcodes different from typical occurrence data. At present GBIF displays data as dots on a map (as do I in http://iphylo.org/~rpage/bold-map/). But barcodes come with a lot more information than that.

This is what phylodiversity looks like

OK, so the title is pure click bait, but here\'s the thing. It seems to
me that the Semantic Web as classically conceived (RDF/XML, SPARQL,
triple stores) has had relatively little impact outside academia,
whereas other technologies such as JSON, NoSQL (e.g., MongoDB, CouchDB)
and graph databases (e.g., Neo4J) have got a lot of developer mindshare.

In biodiversity informatics the Semantic Web has been a round for a
while. We\'ve been pumping out millions of RDF documents (mostly served
by LSIDs) since 2005 and, to a first approximation, nothing has
happened. I\'ve repeatedly blogged about why I think this is (see [this
post for a
summary](http://iphylo.blogspot.co.uk/2015/07/steve-baskauf-on-rdf-and-page-challenge.html)).

I was an early fan of RDF and the Semantic Web, but soon decided that it
was far more hassle than it was worth. The obsession with ontologies,
the problems of globally unique identifiers based on HTTP (http-14
range, anyone?), the need to get a lot of ducks in a row all mad it a
colossal pain. Then I discovered the NoSQL document database
[CouchDB](http://couchdb.apache.org), which is a JSON store that
features map-reduce views rather than on the fly queries. To somebody
with a relational database background this is a bit of a headfuck:

[![Fault
tolerance](https://lh3.googleusercontent.com/-i6RdZ9f7CsU/VnLWdz1uUEI/AAAAAAAAEF0/-BrevKnTYo0/fault-tolerance.png?imgmax=800 "fault-tolerance.png"){style="display:block; margin-left:auto; margin-right:auto;"
border="0" width="460"
height="230"}](http://insidebigdata.com/wp-content/uploads/2014/05/humor_mapreduce.png)

But CouchDB has a great interface, can be replicated to the cloud, and
is FUN (how many times can you say that about a database?). So I
starting playing with CouchDB for small projects, then used it to build
[BioNames](http://bionames.org) and more recently moved
[BioStor](http://biostor.org) to CouchDB hosted both locally and in the
[cloud](http://cloudant.com).

Then there are graph databases such as [Neo4J](http://neo4j.com), which
has some really cool things such as
[GraphGists](http://graphgist.neo4j.com/#!/gists) which is a playground
where you can create interactive graphs and query them ([here\'s an
example I
created](http://iphylo.blogspot.co.uk/2015/08/testing-gbif-taxonomy-with-graph.html)).
Once again, this is FUN.

Another big trend over the last decade is the flight from XML and its
hideous complexities (albeit coupled with great power) to the simplicity
of [JSON](http://json.org) (part of the [rise of
JavaScript](https://github.com/blog/2047-language-trends-on-github)).
JSON makes it very easy to pass around data in simple key-value
documents (with more complexity such as lists if you need them). Pretty
much any modern API will serve you data in JSON.

So, what happened to RDF? Well, along with a plethora of formats (XML,
triples, quads, etc., etc.) it adopted JSON in the form of JSON-LD (see
[JSON-LD and Why I Hate the Semantic
Web](http://manu.sporny.org/2014/json-ld-origins-2/) for background).
JSON-LD lets you have data in JSON (which both people and machines find
easy to understand) and all the complexity/clarity of having the data
clearly labelled using controlled vocabularies such as Dublin Core and
schema.org. This complexity is shunted off into a \"@context\" variable
where it can in many cases be safely ignored.

But what I find really interesting is that instead of JSON-LD being a
way to get developers interested in the rest of the Semantic Web stack
(e.g. HTTP URIs as identifiers, SPARQL, and triple stores), it seems
that what it is really going to do is enable well-described structured
to get access to all the cool things being developed around JSON. For
example, we have document databases such as CouchDB which speaks HTTP
and JSON, and search servers such as
[ElasticSearch](https://www.elastic.co) which make it easy to work with
large datasets. There are also some cool things happening with graph
databases and Javascript, such as
[Hexastore](https://github.com/crubier/Hexastore) (see also Weiss, C.,
Karras, P., & Bernstein, A. (2008, August 1). Hexastore. Proc. VLDB
Endow. VLDB Endowment. <http://doi.org/10.14778/1453856.1453965>, PDF
[here](http://karras.rutgers.edu/hexastore.pdf)) where we create the six
possible indexes of the classic RDF \[subject,predicate,object\] triple
(this is the sort of thing can also be done in CouchDB). Hence we can
have graph databases implemented in a web browser!

So, when we see large-scale \"Semantic Web\" applications that actually
exist and solve real problems, we may well be more likely to see
technologies other than the classic Semantic Web stack. As an example,
see the following paper:

> Szekely, P., Knoblock, C. A., Slepicka, J., Philpot, A., Singh, A.,
> Yin, C., ... Ferreira, L. (2015). Building and Using a Knowledge Graph
> to Combat Human Trafficking. The Semantic Web - ISWC 2015. Springer
> Science + Business Media.
> <http://doi.org/10.1007/978-3-319-25010-6_12>

There\'s a [free PDF
here](http://usc-isi-i2.github.io/papers/szekely15-iswc.pdf), and a
[talk
online](http://videolectures.net/iswc2015_szekely_human_trafficking/).
The consortium behind this project researchers did extensive text
mining, data cleaning and linking, creating a massive collection of
JSON-LD documents. Rather than use a triple store and SPARQL, they
indexed the JSON-LD using ElasticSearch (notice that they generated
graphs for each of the entities they care about, in a sense
denormalising the data).

I think this is likely to be the direction many large-scale projects are
going to be going. Use the Semantic Web ideas of explicit vocabularies
with HTTP URIs for definitions, encode the data in JSON-LD so it\'s
readable by developers (no developers, no projects), then use some of
the powerful (and fun) technologies that have been developed around
semi-structured data. And if you have JSON-LD, then you get SEO for free
by embedding that JSON-LD in your web pages.

In summary, if biodiversity informatics wants to play with the Semantic
Web/linked data then it seems obvious that some combination of JSON-LD
with NoSQL, graph databases, and search tools like ElasticSearch are the
way to go.


OK, so the title is pure click bait, but here's the thing. It seems to me that the Semantic Web as classically conceived (RDF/XML, SPARQL, triple stores) has had relatively little impact outside academia, whereas other technologies such as JSON, NoSQL (e.g., MongoDB, CouchDB) and graph databases (e.g., Neo4J) have got a lot of developer mindshare. In biodiversity informatics the Semantic Web has been a round for a while.

Messaggi di Rogue Scholar

Towards a WikiCite search engine

nodbi: the NoSQL Database Connector

elastic - Elasticsearch for R

This is what phylodiversity looks like

Will JSON, NoSQL, and graph databases save the Semantic Web?