Maëlle Salmon

In the [second post of the series where we obtained data
fromeBird](/blog/2018/08/21/birds-radolfzell/) wedetermined what birds
were observed in the county of Constance, and wecomplemented this
knowledge with some taxonomic and trait information in[the fourth post
of theseries](/blog/2018/09/04/birds-taxo-traits/). Now,we could be
curious about the occurrence of these birds in *scientificwork*. In this
post, we will query the scientific literature and an openscientific data
repository for species names: what have these birds beenstudied for?
Read on if you want to learn how to use R packages allowingto do so!

## Getting a list of 50 species from occurrence data

For more details about the following code, refer to the [previous postof
the series](/blog/2018/08/21/birds-radolfzell/).The single difference is
our adding a step to keep only data for themost recent years.

::: {.highlight}
``` {.chroma tabindex="0"}
# polygon for filteringlandkreis_konstanz <- osmdata::getbb("Landkreis Konstanz",                             format_out = "sf_polygon")crs <- sf::st_crs(landkreis_konstanz)# get and filter dataf_out_ebd <- "ebird/ebd_lk_konstanz.txt"library("magrittr")ebd <- auk::read_ebd(f_out_ebd) %>%  sf::st_as_sf(coords = c("longitude", "latitude"),                 crs = crs) in_indices <- sf::st_within(ebd, landkreis_konstanz)ebd <- dplyr::filter(ebd, lengths(in_indices) > 0)ebd <- as.data.frame(ebd)ebd <- dplyr::filter(ebd, approved, lubridate::year(observation_date) > 2010)
```
:::

For the sake of simplicity, we shall only use the 50 species observedthe
most often.

::: {.highlight}
``` {.chroma tabindex="0"}
species <- ebd %>%  dplyr::count(common_name, sort = TRUE) %>%  head(n = 50) %>%  dplyr::pull(common_name)
```
:::

The species are Carrion Crow, Eurasian Blackbird, Mallard, EurasianCoot,
Great Tit, Great Crested Grebe, Mute Swan, Great Cormorant,Eurasian Blue
Tit, Gray Heron, Black-headed Gull, Common Chaffinch,Common Chiffchaff,
Tufted Duck, European Starling, White Wagtail,European Robin, Little
Grebe, Common Wood-Pigeon, Red-crested Pochard,Ruddy Shelduck, Graylag
Goose, Red Kite, Common Buzzard, EurasianBlackcap, Great Spotted
Woodpecker, Eurasian Magpie, Gadwall, CommonPochard, Eurasian Nuthatch,
Green-winged Teal, House Sparrow, EurasianJay, Yellow-legged Gull,
Yellowhammer, Eurasian Green Woodpecker, EaredGrebe, Eurasian Reed
Warbler, Barn Swallow, Northern Shoveler, EurasianMoorhen, Black
Redstart, Great Egret, White Stork, Eurasian Wren,Long-tailed Tit,
Common House-Martin, Eurasian Kestrel, EuropeanGoldfinch and European
Greenfinch[(`glue::glue_collapse(species, sep = ", ", last = " and ")`)](https://twitter.com/LucyStats/status/1031938964796657665?s=19).

## Querying the scientific literature

Just like rOpenSci has a taxonomic
toolbelt([`taxize`](https://github.com/ropensci/taxize)) and a
speciesoccurrence data toolbelt
([`spocc`](https://github.com/ropensci/spocc)),it has a super package
for querying the scientific
literature:[`fulltext`](https://github.com/ropensci/fulltext)! This
packagesupports search for "PLOS via the rplos package, Crossref via
thercrossref package, Entrez via the rentrez package, arXiv via the
aRxivpackage, and BMC, Biorxiv, EuroPMC, and Scopus via internal
helperfunctions".

We shall use `fulltext` to retrieve the titles and abstracts
ofscientific articles mentioning each species, and will use `tidytext`
tocompute the most prevalent words in these works.

We first define a function retrieving the titles and abstracts of
worksobtained as result when querying one species name.

We use `dplyr::bind_rows` because we want all results for one species
atonce, while `fulltext` returns a list of data.frames with one
data.frameby data source.

::: {.highlight}
``` {.chroma tabindex="0"}
.get_papers <- function(species){  species %>%    tolower() %>%    fulltext::ft_search() %>%    fulltext::ft_get() %>%    fulltext::ft_collect() %>%    fulltext::ft_chunks(c("title", "abstract")) %>%    fulltext::ft_tabularize() %>%    dplyr::bind_rows()}.get_papers(species[1]) %>%  dplyr::pull(title)
```
:::

    ##  [1] "Great spotted cuckoo nestlings have no antipredatory effect on magpie or carrion crow host nests in southern Spain"                                ##  [2] "Donor-Control of Scavenging Food Webs at the Land-Ocean Interface"                                                                                 ##  [3] "Formal comment to Soler et al.: Great spotted cuckoo nestlings have no antipredatory effect on magpie or carrion crow host nests in southern Spain"##  [4] "Socially Driven Consistent Behavioural Differences during Development in Common Ravens and Carrion Crows"                                          ##  [5] "Behavioral Responses to Inequity in Reward Distribution and Working Effort in Crows and Ravens"                                                    ##  [6] "Early Duplication of a Single MHC IIB Locus Prior to the Passerine Radiations"                                                                     ##  [7] "Investigating the impact of media on demand for wildlife: A case study of Harry Potter and the UK trade in owls"                                   ##  [8] "New Caledonian Crows Rapidly Solve a Collaborative Problem without Cooperative Cognition"                                                          ##  [9] "Nest Predation Deviates from Nest Predator Abundance in an Ecologically Trapped Bird"                                                              ## [10] "Dietary Compositions and Their Seasonal Shifts in Japanese Resident Birds, Estimated from the Analysis of Volunteer Monitoring Data"

If we were working on a scientific study, we'd add a few more
filters,e.g. having the species mentioned in the abstract, and not
onlysomewhere in the paper which is probably the way the
differentliterature search providers define a match. But we're not, so
we cankeep our query quite free! My favourite paper involving the
Carrion Crowis ["Investigating the impact of media on demand for
wildlife: A casestudy of Harry Potter and the UK trade
inowls"](http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0182368)because
it's a fun and important scientific question, and is supportedby open
data (by the way you can access CITES trade data (internationaltrade in
endangered species) in R
using[`cites`](https://github.com/ecohealthalliance/cites/) and
CITESSpeciesplus database
using[`rcites`](https://docs.ropensci.org/rcites//)).

We then apply this function to all 50 species and keep each article
onlyonce.

::: {.highlight}
``` {.chroma tabindex="0"}
get_papers <- ratelimitr::limit_rate(.get_papers,                                     rate = ratelimitr::rate(1, 2))all_papers <- purrr::map_df(species, get_papers)nrow(all_papers)
```
:::

    ## [1] 522

::: {.highlight}
``` {.chroma tabindex="0"}
all_papers <- unique(all_papers)nrow(all_papers)
```
:::

    ## [1] 378

Now, we get the most common words from titles and abstracts. For that
wefirst append the title to the abstract which is a quick hack.

::: {.highlight}
``` {.chroma tabindex="0"}
library("tidytext")library("rcorpora")stopwords <- corpora("words/stopwords/en")$stopWordsall_papers %>%  dplyr::group_by(title, abstract) %>%  dplyr::summarise(text = paste(title, abstract)) %>%  dplyr::ungroup() %>%  unnest_tokens(word, text) %>%  dplyr::filter(!word %in% stopwords) %>%  dplyr::count(word, sort = TRUE) -> words
```
:::

So, what are the most common words in these papers?

::: {.highlight}
``` {.chroma tabindex="0"}
head(words, n = 10) 
```
:::

    ##           word   n## 1      species 754## 2        birds 514## 3        virus 270## 4        avian 268## 5         bird 262## 6        study 243## 7     breeding 231## 8         wild 227## 9  populations 217## 10  population 213

Not too surprising, and obviously less entertaining than looking
atindividual species' results. Maybe a wordcloud can give us a better
ideaof the wide area of topics of studies involving our 50 most
frequentbird species. We use the
[`wordcloud`package](https://cran.r-project.org/web/packages/wordcloud/index.html).

::: {.highlight}
``` {.chroma tabindex="0"}
library("wordcloud")with(words, wordcloud(word, n, max.words = 100))
```
:::

![wordcloud of titles and abstracts of
scientificpapers](/img/blog-images/2018-09-11-birds-science/wordcloud-1.png)

We see that topics include ecological words such as "foraging" but
alsoepidemiological questions since "influenza" and "h5n1" come up. Now,
howinformative as this wordcloud can be, it's a bit ugly, so we'll
prettifyit using the
[`wordcloud2`package](https://github.com/Lchiffon/wordcloud2) instead,
and thesilhouette of a bird
[fromPhylopic](http://phylopic.org/image/6209c9be-060e-4d7f-bc74-a75f3ccf4629/).

::: {.highlight}
``` {.chroma tabindex="0"}
bird <- words %>%  head(n = 100) %>%  wordcloud2::wordcloud2(figPath = "bird.png",                        color = "black", size = 1.5)# https://www.r-graph-gallery.com/196-the-wordcloud2-library/htmlwidgets::saveWidget(bird,                        "tmp.html",                        selfcontained = F)
```
:::

I wasn't able to `webshot` the resulting html despite increasing
the`delay` parameter so I screenshot it by hand!

::: {.highlight}
``` {.chroma tabindex="0"}
magick::image_read("screenshot.png")
```
:::

<figure>
<img src="/img/blog-images/2018-09-11-birds-science/wordcloud2-1.png"
width="1366" alt="wordcloud shaped as a bird" />
<figcaption aria-hidden="true"><p>wordcloud shaped as a
bird</p></figcaption>
</figure>

The result is a bit kitsch, doesn't include the word "species", oneneeds
to know it's the silhouette of a bird to recognize it, and we'dneed to
work a bit on not reshaping the silhouette, but it's fun as itis.

## Querying scientific open data

There are quite a few scientific open data repositories out there,
amongwhich the giant [DataONE](https://www.dataone.org/) that has an
APIinterfaced with an R package. We shall use it to perform a
searchsimilar to the previous section, but looking at the data indexed
onDataONE. Since DataONE specializes in ecological and environmental
data,we expect to find rather ecological data.

We first define a function to retrieve metadata of datasets for
onespecies. It looks the species names in the abstract.

::: {.highlight}
``` {.chroma tabindex="0"}
.get_meta <- function(species){    cn <- dataone::CNode("PROD")  search <- list(q = glue::glue("abstract:{species}"),                        fl = "id,title,abstract",                        sort = "dateUploaded+desc")    result <- dataone::query(cn, solrQuery = search,                           as="data.frame")    if(nrow(result) == 0){    NULL  }else{    # otherwise one line by version  result <- unique(result)    tibble::tibble(species = species,                 title = result$title,                 abstract = result$abstract)  }}
```
:::

Note that DataONE searching could be more precise: one can choose
tosearch from a given data source only for instance. See the
[searchingDataONEvignette](https://github.com/DataONEorg/rdataone/blob/master/vignettes/searching-dataone.Rmd).

::: {.highlight}
``` {.chroma tabindex="0"}
get_meta <- ratelimitr::limit_rate(.get_meta,                                     rate = ratelimitr::rate(1, 2))all_meta <- purrr::map_df(species, get_meta)nrow(all_meta)
```
:::

    ## [1] 266

::: {.highlight}
``` {.chroma tabindex="0"}
length(unique(all_meta$species))
```
:::

    ## [1] 35

35 species are represented.

::: {.highlight}
``` {.chroma tabindex="0"}
all_meta <- unique(all_meta[,c("title", "abstract")])nrow(all_meta)
```
:::

    ## [1] 104

We then extract the most common words.

::: {.highlight}
``` {.chroma tabindex="0"}
all_meta %>%  dplyr::group_by(title, abstract) %>%  dplyr::summarise(text = paste(title, abstract)) %>%  dplyr::ungroup() %>%  unnest_tokens(word, text) %>%  dplyr::filter(!word %in% stopwords) %>%  dplyr::count(word, sort = TRUE) -> data_wordshead(data_words, n = 10)
```
:::

    ## # A tibble: 10 x 2##    word           n##    <chr>      <int>##  1 data         153##  2 species      120##  3 birds         94##  4 breeding      87##  5 feeding       75##  6 population    65##  7 bird          60##  8 genetic       58##  9 study         56## 10 effects       54

Data is the most common word which is quite logical for metadata
ofactual datasets. Let's also have a look at a regular wordcloud.

::: {.highlight}
``` {.chroma tabindex="0"}
with(data_words, wordcloud(word, n, max.words = 100))
```
:::

![wordcloud of titles and abstracts of
scientificmetadata](/img/blog-images/2018-09-11-birds-science/wordcloud3-1.png)

As expected, the words seem more focused on ecology than when looking
atscientific papers. DataONE is a gigantic data catalogue, where one
could

- study the results of such queries (e.g. meta studies of number of,say,
  versions by datasets)

- or find data to integrate to a new study. If you want to
  *download*data from DataONE, refer to the [download
  datavignette](https://github.com/DataONEorg/rdataone/blob/master/vignettes/download-data.Rmd).

## Conclusion

In this post, we used the rOpenSci `fulltext` package, and the
DataONE`dataone` package, to search for bird species names in scientific
papersand scientific open datasets. We were able to draw
wordcloudsrepresenting the diversity of topics of studies in which the
birds hadbeen mentioned or studied. Such a search could be fun to do for
yourfavourite bird(s)! And in general, following the same approach you
couldanswer your own specific research question.

### Scientific literature access

As a reminder, the pipeline to retrieve abstracts and titles of
worksmentioning a bird species was quite smooth:

::: {.highlight}
``` {.chroma tabindex="0"}
species %>%    tolower() %>%    fulltext::ft_search() %>%    fulltext::ft_get() %>%    fulltext::ft_collect() %>%    fulltext::ft_chunks(c("title", "abstract")) %>%    fulltext::ft_tabularize() %>%    dplyr::bind_rows()
```
:::

`fulltext` gives you a lot of power! Other rOpenSci accessing
literaturedata include
[`europepmc`](https://github.com/ropensci/europepmc), RInterface to
Europe PMC RESTful Web
Service;[`jstor`](https://github.com/ropensci/jstor);[`suppdata`](https://github.com/ropensci/suppdata)
for extractingsupplemental information, and [muchmore](/packages/).

### Scientific data access... and publication with R

In this post we used the
[`dataone`package](https://github.com/DataONEorg/rdataone) to access
data fromDataONE. That same package allows uploading data to DataONE.
TherOpenSci suite features
the[`rfigshare`](https://github.com/ropensci/rfigshare) package for
gettingdata from, and publishing data to,
[Figshare](https://figshare.com/).For preparing your own data and its
documentation for publication, checkout the [`EML`
package](https://github.com/ropensci/EML) for writingmetadata respecting
the Ecological Metadata Standard, and the [unconf`dataspice`
project](https://github.com/ropenscilabs/dataspice) forsimpler metadata
entry.

Explore more of our packages suite, including and beyond access
toscientific literature &data and data publication,[here](/packages/).

### No more birding? No, your turn!

This was the last post of this series, that hopefully provided
anoverview of how rOpenSci packages can help you learn more about
birds,and can support your workflow. As a reminder, in this series we
saw

- [How to identify spots for birding using open
  geographicaldata](/blog/2018/08/14/where-to-bird/).Featuring
  `opencage` for geocoding, `bbox` for bounding boxcreation, `osmdata`
  for OpenStreetMap's Overpass API querying,`osmplotr` for map drawing
  using OpenStreetMap's data.

- [How to obtain bird occurrence data
  inR](/blog/2018/08/21/birds-radolfzell/).Featuring `rebird` for
  interaction with the eBird's API, and `auk`for munging of the whole
  eBird dataset.

- [How to extract text from old natural
  historydrawings](/blog/2018/08/28/birds-ocr/).Featuring `magick` for
  image manipulation, `tesseract` for OpticalCharacter Recognition,
  `cld2` and `cld3` for language detection, and`taxize::gnr_resolve` for
  taxonomic name resolution.

- [How to complement an occurrence dataset with taxonomy and
  traitinformation](/blog/2018/09/04/birds-taxo-traits/).Featuring
  `taxize`, taxonomic toolbelt for R, and `traits`,providing access to
  species traits data.

- How to query the scientific literature and scientific open
  datarepositories. This is the post you've just read!

That's a wrap! But now, don't *you* hesitate to explore our
packagessuite for your own needs, and to share about your use cases of
rOpenScipackages as a birder or not via [our friendly
discussionforum](https://discuss.ropensci.org/c/usecases)! Happy
birding!


In the second post of the series where we obtained data fromeBird wedetermined what birds were observed in the county of Constance, and wecomplemented this knowledge with some taxonomic and trait information inthe fourth post of theseries.

What have these birds been studied for? Querying science outputs with R

Thanks to the [first post of theseries](/blog/2018/08/14/where-to-bird/)
we knowwhere to observe birds near Radolfzell's Max Planck Institute
forOrnithology, so we could go and do that! Or we can stay behind
ourlaptops and take advantage of [eBird](https://ebird.org/home),
afantastic bird sightings aggregator! As explained by Matt
Strimas-Mackeyin [his recent blog post](/blog/2018/08/07/auk/),"The
eBird database currently contains over 500 million records of
birdsightings, spanning every country and over 98% of species, making it
anextremely valuable resource for bird research and conservation.".

Luckily for us, there are no less than *two* rOpenSci packages giving
usaccess to eBird data! In this blog post, I shall play with both of
them,highlighting their respective strengths, while discovering what
birdsare observed in the area.

## How to access eBird data?

There are two ways to access eBird data with an R package for each
ofthese methods,

- whole dataset download via `auk` maintained by
  [MattStrimas-Mackey](http://strimas.com/),

- APIs, via `rebird` maintained by
  [SebastianPardo](https://sebpardo.github.io/).

Your use case will help you decide which entry point is the
mostappropriate for your use case. *Note that both packages have
documentedtheir respective applications in order to help potential
users:[`rebird`
README](https://github.com/ropensci/rebird#auk-vs-rebird),[`auk`README](https://github.com/CornellLabofOrnithology/auk#auk-vs-rebird).*

- You want to study a region, or a bird, quite deeply and you evenwant
  absence/presence data, not only presence data. Use `auk`!

- You want to build a tool based on recent observations only or youwant
  to get a quick taste of eBird's data. Use `rebird`!

- A bit provocatively, do you want birds data only? If not, maybeyou'll
  need a combination of `auk`/`rebird` and another package.Check out
  this list of [data providers covered
  by`spocc`](https://github.com/ropensci/spocc#spocc), umbrella
  packagefor rOpenSci's packages accessing occurrence data. Many data
  sourcesactually end up in
  [GBIFdatasets](https://www.gbif.org/dataset/search), eBird seems
  toupload their data there once a year.

- You want to analyze *your* eBird's sightings? Check out
  thework-in-progress [`myebird`](https://github.com/sebpardo/myebird)
  by[Sebastian Pardo](https://sebpardo.github.io/),
  `rebird`'smaintainer, and [this app](http://www.ebirdedgeapp.io/) by
  [SimónValdez-Juarez](https://www.sfu.ca/biology/people/profiles/svaldez.html)and
  [Sebastian Pardo](https://sebpardo.github.io/) highlighting themost
  endangered species you observed.

- You're writing a birder's guide to rOpenSci? Use both `rebird`
  and`auk` to show them off!

## How to get access to eBird's data

### Whole eBird dataset, quarterly updated

One needs to first [create an
eBirdaccount](https://secure.birds.cornell.edu/cassso/login) and
[thenrequest access to the data](https://ebird.org/data/request). Once
onehas gotten green light from eBird (in my case a few days following
myrequest), after a small dance of joy it's time to head to
eBird's[download page](https://ebird.org/data/download/ebd). If one
doesn'twant nor need to download the whole eBird Basic Dataset (EBD),
one canrequest a custom download, which I did, asking for only the data
forGermany which I got after a few days (the time to receiving the link
todownload a custom dataset is variable). While waiting, I worked on
the`rebird` part of this post, among other things.

### API key? Not yet

At the moment, `rebird` interfaces the version 1.1 eBird APIs that
willbe retired "at some point in thefuture".When this happens, the
`rebird` package [will use the
newAPI](https://github.com/ropensci/rebird/issues/58) which will
meanyou'll need an API key. Currently, though, you don't need
anyauthentication to use `rebird`.

## Using rebird while waiting for the eBird's full dataset

In the following, we'll use the rOpenSci's package `rebird` to get
andmap all observations in the last 30 days near Radolfzell in Germany.

The *Radolfzell* part of that sentence is a bit different than in
[thelast post about finding bird hides near the MPI institute
forornithology](/blog/2018/08/14/where-to-bird/): Iwant all observations
inside the polygon of the district of Constance(Landkreis Konstanz,
including Radolfzell... and a protected naturalarea!) so I'll first need
to get it. For doing that I'll use`osmdata::getbb`, that uses the free
Nominatim API provided byOpenstreetmap.

::: {.highlight}
``` {.chroma tabindex="0"}
library("sf")landkreis_konstanz <- osmdata::getbb("Landkreis Konstanz",                             format_out = "sf_polygon")plot(landkreis_konstanz)
```
:::

![Limits of the County
ofConstance](/img/blog-images/2018-08-21-birds-radolfzell/lk-1.png)

Neither `rebird` nor `spocc` currently offer built-in trimming
ofoccurrence data to a polygon (whereas `osmdata` does). A
furtherdifficulty created by eBird's API is that it doesn't allow for
the useof a bounding box, but instead demands a `lat`, `lng` and a
`dist`defining the radius of interest from given `lat`/`lng` in
kilometers.Thanks to [Marco Sciaini](https://marcosci.rbind.io/) for
providing mewith an easy way to compute `dist`, using the `sf` package.

::: {.highlight}
``` {.chroma tabindex="0"}
coord <- sf::st_coordinates(landkreis_konstanz)bbox <- c(x1 = min(coord[, "X"]),          x2 = max(coord[, "X"]),          y1 = min(coord[, "Y"]),          y2 = max(coord[, "Y"]))center <- c(x = (bbox["x1"] + bbox["x2"])/2,            y = (bbox["y1"] + bbox["y2"])/2)dist <- landkreis_konstanz %>%  sf::st_cast("POINT") %>%  sf::st_distance() %>%   max() * 0.5dist
```
:::

    ## 24129.15 m

Now, we can make the query.

::: {.highlight}
``` {.chroma tabindex="0"}
birds <- rebird::ebirdgeo(species = NULL,                          lng = center["x.x1"],                          lat = center["y.y1"],                          back = 30,                          dist = as.numeric(                            units::set_units(dist, "km")))nrow(birds)
```
:::

    ## [1] 55

::: {.highlight}
``` {.chroma tabindex="0"}
str(birds)
```
:::

    ## Classes 'tbl_df', 'tbl' and 'data.frame':    55 obs. of  12 variables:##  $ lng            : num  8.94 8.94 8.94 8.94 8.94 ...##  $ locName        : chr  "Radolfzeller Aachmündung (Bodensee)" "Radolfzeller Aachmündung (Bodensee)" "Radolfzeller Aachmündung (Bodensee)" "Radolfzeller Aachmündung (Bodensee)" ...##  $ sciName        : chr  "Chroicocephalus ridibundus" "Motacilla alba" "Rallus aquaticus" "Aythya fuligula" ...##  $ obsValid       : logi  TRUE TRUE TRUE TRUE TRUE TRUE ...##  $ locationPrivate: logi  FALSE FALSE FALSE FALSE FALSE FALSE ...##  $ obsDt          : chr  "2018-08-08 13:30" "2018-08-08 13:30" "2018-08-08 13:30" "2018-08-08 13:30" ...##  $ obsReviewed    : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...##  $ comName        : chr  "Black-headed Gull" "White Wagtail" "Water Rail" "Tufted Duck" ...##  $ lat            : num  47.7 47.7 47.7 47.7 47.7 ...##  $ locID          : chr  "L3314048" "L3314048" "L3314048" "L3314048" ...##  $ locId          : chr  "L3314048" "L3314048" "L3314048" "L3314048" ...##  $ howMany        : int  NA 2 1 1 NA 3 NA NA 8 20 ...

Now that we have the occurrence data, let's plot it to see
whethertrimming is required.

::: {.highlight}
``` {.chroma tabindex="0"}
crs <- sf::st_crs(landkreis_konstanz)birds_sf <- sf::st_as_sf(birds,                         coords = c("lng", "lat"),                          crs = crs)library("ggplot2")ggplot() +  geom_sf(data = landkreis_konstanz) +  geom_sf(data = birds_sf) +  theme(legend.position = "bottom") +  hrbrthemes::theme_ipsum() +  ggtitle("eBird observations over the last 30 days",          subtitle = "Observations within a circle around the County of Constance")
```
:::

![map of raw observations within
acircle](/img/blog-images/2018-08-21-birds-radolfzell/map_circle-1.png)

Yes, trimming is required! It'd have been too bad not to learn how to
doit, anyway. We also add the MPI to the map.

::: {.highlight}
``` {.chroma tabindex="0"}
# which parts of the oject are in the countyin_indices <- sf::st_within(birds_sf, landkreis_konstanz)# filter themtrimmed_birds <- dplyr::filter(birds_sf,                               lengths(in_indices) > 0)# summarize to get no. of birds by  locationsummarized_birds <- trimmed_birds %>%  dplyr::group_by(locName) %>%  dplyr::summarise(n = n())# MPI mpi <- opencage::opencage_forward("Am Obstberg 1 78315 Radolfzell",                                   limit = 1)$resultscoords <- data.frame(lon = mpi$geometry.lng,                     lat = mpi$geometry.lat)crs <- sf::st_crs(landkreis_konstanz)mpi_sf <- sf::st_as_sf(coords,                       coords = c("lon", "lat"),                        crs = crs)# Map!ggplot() +  geom_sf(data = landkreis_konstanz) +  geom_sf(data = summarized_birds,          aes(size = n), show.legend = "point") +  hrbrthemes::theme_ipsum() +  ggtitle("eBird observations over the last 30 days",          subtitle = "County of Constance, MPI as a triangle") +  geom_sf(data = mpi_sf,          shape = 2) 
```
:::

![trimmed observations in
thecounty](/img/blog-images/2018-08-21-birds-radolfzell/trimmed1-1.png)

We got 49 observations (`nrow(trimmed_birds)`) of 49
species(`length(unique(trimmed_birds$comName))`), over 2
places(`length(unique(trimmed_birds$locName))`) during 5 observation
sessions.Hopefully merely an appetizer to what we can get from using the
fulleBird dataset in the next section...

Note that the initial query could have been made with `spocc` whichwould
have helped using the rOpenSci occurrence suite.

::: {.highlight}
``` {.chroma tabindex="0"}
birds2 <- spocc::occ(from = "ebird",                     ebirdopts = list(method = "ebirdgeo",                                      species = NULL,                                      lng = center["x.x1"],                                      lat = center["y.y1"],                                      back = 30,                                      dist = as.numeric(                                        units::set_units(dist, "km"))))                     mapr::map_leaflet(birds2)
```
:::

<figure>
<img src="/img/blog-images/2018-08-21-birds-radolfzell/mapr-1.png"
alt="mapr leaflet map of observations locations" />
</figure>

Quite handy!

Now, let's explore the whole eBird dataset for Germany.

## Using auk to process EBD dataset for Germany

After getting access to a custom dataset corresponding to the EBD
forGermany only, I used `auk`'s documentation and
[thispost](/blog/2018/08/07/auk/) to learn how to processit. Since I
wasn't planning on zero-filling the data to getpresence/absence counts,
I was able to ignore the sampling event datathat contains the
checklist-level information (e.g. time and date,location, and search
effort information). For an example of a moreadvanced `auk` workflow
involving the **full** EBD, and sampling data,refer to [Matt
Strimas-Mackey's own blog post about hispackage](/blog/2018/08/07/auk/).

### Preparing the dataset

Here, the workflow is to *clean* the data and to *filter* it using oneof
`auk`'s built-in filters and then polygon filtering as earlier inthis
post. All steps are quite fast, because the custom dataset forGermany
isn't too big (a few hundred megabytes).

Cleaning happens in the following:

::: {.highlight}
``` {.chroma tabindex="0"}
ebd_dir <- "C:/Users/Maelle/Documents/ropensci/ebird"f <- file.path(ebd_dir, "ebd_DE_relMay-2018.txt")f_clean <- file.path(ebd_dir, "ebd_DE_relMay-2018_clean.txt")auk::auk_clean(f, f_out = f_clean, remove_text = TRUE)
```
:::

Then one can filter the data. Note that the `auk_extent` function
thatonly retains observations within a bounding box has been
renamed`auk_bbox` in the dev version of `auk`, the old name will be
deprecatedsoon.

::: {.highlight}
``` {.chroma tabindex="0"}
ebd_dir <- "C:/Users/Maelle/Documents/ropensci/ebird"f_in_ebd <- file.path(ebd_dir, "ebd_DE_relMay-2018_clean.txt")library("magrittr")landkreis_konstanz_coords <- sf::st_coordinates(landkreis_konstanz)ebd_filter <- auk::auk_ebd(f_in_ebd) %>%   auk::auk_extent(c(min(landkreis_konstanz_coords[, "X"]),                    min(landkreis_konstanz_coords[, "Y"]),                     max(landkreis_konstanz_coords[, "X"]),                     max(landkreis_konstanz_coords[, "Y"])))ebd_filter
```
:::

    ## Input ##   EBD: C:\Users\Maelle\Documents\ropensci\ebird\ebd_DE_relMay-2018_clean.txt ## ## Output ##   Filters not executed## ## Filters ##   Species: all##   Countries: all##   States: all##   BCRs: all##   Spatial extent: Lon 8.6 - 9.2; Lat 47.7 - 47.9##   Date: all##   Start time: all##   Last edited date: all##   Protocol: all##   Project code: all##   Duration: all##   Distance travelled: all##   Records with breeding codes only: no##   Complete checklists only: no

::: {.highlight}
``` {.chroma tabindex="0"}
fs::dir_create("ebird")f_out_ebd <- "ebird/ebd_lk_konstanz.txt"f_out_sampling <- "ebird/ebd_lk_konstanz_sampling.txt"ebd_filtered <- auk::auk_filter(ebd_filter, file = f_out_ebd,                                overwrite = TRUE)
```
:::

On top of this filtering with `auk`, after loading the data we
filterobservations inside the polygon of the county.

::: {.highlight}
``` {.chroma tabindex="0"}
crs <- sf::st_crs(landkreis_konstanz)ebd <- auk::read_ebd(f_out_ebd) %>%  sf::st_as_sf(coords = c("longitude", "latitude"),                 crs = crs) in_indices <- sf::st_within(ebd, landkreis_konstanz)ebd <- dplyr::filter(ebd, lengths(in_indices) > 0)ebd <- as.data.frame(ebd)
```
:::

### What are the observed birds?

Before looking at species names, let's have a brief look at the size
andtemporal extent of the data.

::: {.highlight}
``` {.chroma tabindex="0"}
library("ggplot2")dim(ebd)
```
:::

    ## [1] 10156    41

::: {.highlight}
``` {.chroma tabindex="0"}
ebd %>%  dplyr::mutate(year = lubridate::year(observation_date)) %>%ggplot() +  geom_bar(aes(year))  +  hrbrthemes::theme_ipsum(base_size = 12, axis_title_size = 12, axis_text_size = 12) +  ylab("No. of eBird observations") +  xlab("Time (years)") +  ggtitle("Full eBird dataset for the County of Constance")
```
:::

![No. of eBird observations over
theyears](/img/blog-images/2018-08-21-birds-radolfzell/barplot-1.png)

eBird started in 2002 but only became global in 2010. It allows peopleto
enter older observations, though.

Now we can look at what birds have been reported the most.

::: {.highlight}
``` {.chroma tabindex="0"}
ebd %>%  dplyr::filter(approved) %>%  dplyr::count(scientific_name, common_name) %>%  dplyr::arrange(- n) %>%  head(n = 10) %>%  knitr::kable()
```
:::

| scientific_name            | common_name         |   n |
|:---------------------------|:--------------------|----:|
| Corvus corone              | Carrion Crow        | 288 |
| Turdus merula              | Eurasian Blackbird  | 285 |
| Anas platyrhynchos         | Mallard             | 273 |
| Fulica atra                | Eurasian Coot       | 268 |
| Parus major                | Great Tit           | 266 |
| Podiceps cristatus         | Great Crested Grebe | 254 |
| Ardea cinerea              | Gray Heron          | 236 |
| Cygnus olor                | Mute Swan           | 234 |
| Cyanistes caeruleus        | Eurasian Blue Tit   | 233 |
| Chroicocephalus ridibundus | Black-headed Gull   | 223 |

I had to google most of them, but only because I didn't know
thescientific and English names of these birds: they're birds even I,
not abirder, know, probably because they're also common in Brittany
where Igrew up.

We can also look at birds whose observation was rejected. Out of
10156observations only 64 were reviewed, and only 5 were not approved.

::: {.highlight}
``` {.chroma tabindex="0"}
ebd %>%  dplyr::select(scientific_name, common_name,               approved, reviewed, reason) %>%  dplyr::filter(!approved) %>%  knitr::kable()
```
:::

| scientific_name     | common_name       | approved | reviewed | reason                    |
|:--------------------|:------------------|:---------|:---------|:--------------------------|
| Cygnus atratus      | Black Swan        | FALSE    | TRUE     | Species-Introduced/Exotic |
| Cygnus atratus      | Black Swan        | FALSE    | TRUE     | Species-Introduced/Exotic |
| Cygnus atratus      | Black Swan        | FALSE    | TRUE     | Species-Introduced/Exotic |
| Oxyura leucocephala | White-headed Duck | FALSE    | TRUE     | Species-Introduced/Exotic |
| Mareca sibilatrix   | Chiloe Wigeon     | FALSE    | TRUE     | Species-Introduced/Exotic |

Black Swans [are mostly present in Australia, imported and escaped in
afew other places](https://en.wikipedia.org/wiki/Black_swan) but
eBirdmostly doesn't accept the entry of exotic species [although
it'sdebated](https://help.ebird.org/customer/portal/questions/5390335-exclusion-of-not-native-birds-in-lists).In
any case, eBird's curation of the data entered is quite admirable.

### Who observed birds?

In [one of his latest
blogposts](https://recology.info/2018/06/butte-county/) Scott
Chamberlainmentioned the legendary Lowell Ahart, super plant collector
in ButteCounty, California. Does the county of Constance have a super
birder?

::: {.highlight}
``` {.chroma tabindex="0"}
(first_birder <- ebd %>%  dplyr::count(observer_id) %>%  dplyr::arrange(- n) %>%  head(n = 1) )
```
:::

    ## # A tibble: 1 x 2##   observer_id     n##   <chr>       <int>## 1 obsr457108   3551

::: {.highlight}
``` {.chroma tabindex="0"}
(proportion <- round(first_birder$n/nrow(ebd),                    digits = 2))
```
:::

    ## [1] 0.35

Wow, that person made 35% of eBird observations in the county! The EBDno
longer provides names (consequence of the [EU General Data
ProtectionRegulation](https://www.eugdpr.org/)) but from the checklist
ID one canget access to the checklist page e.g
[thisone](https://ebird.org/view/checklist/S42391392) where the name of
theobserver is present. The super birder of the County of Constance
is[Antonio Anta Bink](https://ebird.org/profile/NDU3MTA4/DE).

## Conclusion

### R packages for occurrence data

In this post I gave a rough view of what birds are present in the
countyaround Radolfzell: Eurasian Blackbirds, Carrion Crows, Great
Tits... butnot Black Swans in eBird's data. We mostly illustrated the
use of two Rpackages accessing eBird's data:

- `auk` for processing the gigantic whole eBird's dataset.

- `rebird` for getting access to recent data via an API. `rebird` ispart
  of a larger collection of packages for occurrence data
  withinrOpenSci's suite, with `spocc` being an umbrella package
  accessingseveral data sources; `scrubr` a helper for cleaning data
  obtainedthis way; and `mapr` a utility package for mapping such data.

Explore these packages, and more of rOpenSci's suite, by checking outour
[packages page](/packages/)!

### More birding soon!

Stay tuned for the next post in this series, that'll mark a break
frommodern data since we'll try to extract information from old
naturalhistory bird drawings! After that, in a following post we'll come
backto the occurrence data obtained from eBird in order to complement
itwith open taxonomic and traits data. In the meantime, happy
(e)birding!


Thanks to the first post of theseries we knowwhere to observe birds near Radolfzell’s Max Planck Institute forOrnithology, so we could go and do that! Or we can stay behind ourlaptops and take advantage of eBird, afantastic bird sightings aggregator!

Messaggi di Rogue Scholar

What have these birds been studied for? Querying science outputs with R

What birds are observed near Radolfzell? Bird occurrence data in R