Jeroen Ooms

Scientific articles are typically locked away in PDF format, a format
designed primarily for printing but not so great for searching or
indexing. The new
[pdftools](https://cran.r-project.org/package=pdftools) package allows
for extracting text and metadata from pdf files in R. From the extracted
plain-text one could find articles discussing a particular drug or
species name, without having to rely on publishers providing metadata,
or pay-walled search engines.

The pdftools slightly overlaps with the
[Rpoppler](https://cran.r-project.org/web/packages/Rpoppler/index.html)
package by Kurt Hornik. The main motivation behind developing pdftools
was that Rpoppler depends on glib, which does not work well on Mac and
Windows. The pdftools package uses the poppler c++ interface together
with Rcpp, which results in a lighter and more portable implementation.

## Installing pdftools

[[cran]{.label
.label-warning}](https://cran.rstudio.com/web/packages/pdftools/)
[[github]{.label .label-info}](https://github.com/ropensci/pdftools)

On Windows and Mac the binary packages can be installed directly from
CRAN:

::: {.highlight}
``` {.chroma tabindex="0"}
install.packages("pdftools")
```
:::

Installation on Linux requires the poppler development library. On
Debian/Ubuntu:

``` {tabindex="0"}
sudo apt-get install libpoppler-cpp-dev
```

On Fedora or CentOS:

``` {tabindex="0"}
sudo yum install poppler-cpp-devel
```

If you want to install the package from source on Mac OS-X you need
brew:

``` {tabindex="0"}
brew install poppler
```

That's it.

## Getting started

The `?pdftools` manual page shows a brief overview of the main
utilities. The most important function is `pdf_text` which returns a
character vector of length equal to the number of pages in the pdf. Each
string in the vector contains a plain text version of the text on that
page.

::: {.highlight}
``` {.chroma tabindex="0"}
library(pdftools)download.file("http://arxiv.org/pdf/1403.2805.pdf", "1403.2805.pdf", mode = "wb")txt <- pdf_text("1403.2805.pdf")# first page textcat(txt[1])# second page textcat(txt[2])
```
:::

In addition, the package has some utilities to extract other data from
the PDF file. The `pdf_toc` function shows the table of contents, i.e.
the section headers which pdf readers usually display in a menu on the
left. It looks pretty in JSON:

::: {.highlight}
``` {.chroma tabindex="0"}
# Table of contentstoc <- pdf_toc("1403.2805.pdf")# Show as JSONjsonlite::toJSON(toc, auto_unbox = TRUE, pretty = TRUE)
```
:::

Other functions provide information about fonts, attachments and
metadata such as the author, creation date or tags.

::: {.highlight}
``` {.chroma tabindex="0"}
# Author, version, etcinfo <- pdf_info("1403.2805.pdf")# Table with fontsfonts <- pdf_fonts("1403.2805.pdf")
```
:::

## Bonus feature: rendering pdf

A bonus feature on most platforms is rendering of PDF files to bitmap
arrays. The poppler library provides all functionality to implement a
complete PDF reader, including graphical display of the content. In R we
can use pdf_render_page to render a page of the PDF into a bitmap, which
can be stored as e.g. png or jpeg.

::: {.highlight}
``` {.chroma tabindex="0"}
# renders pdf to bitmap arraybitmap <- pdf_render_page("1403.2805.pdf", page = 1)# save bitmap imagepng::writePNG(bitmap, "page.png")jpeg::writeJPEG(bitmap, "page.jpeg")webp::write_webp(bitmap, "page.webp")
```
:::

This feature now works on all platforms.

## Limitations

Data scientists are often interested in data from tables. Unfortunately
the pdf format is pretty dumb and does not have notion of a table
(unlike for example HTML). Tabular data in a pdf file is nothing more
than strategically positioned lines and text, which makes it difficult
to extract the raw data.

::: {.highlight}
``` {.chroma tabindex="0"}
txt <- pdf_text("http://arxiv.org/pdf/1406.4806.pdf")# some tablescat(txt[18])cat(txt[19])
```
:::

Pdftools usually does a decent job in retaining the positioning of table
elements when converting from pdf to text. But the output is still very
dependent on the formatting of the original pdf table, which makes it
very difficult to write a generic table extractor. But with a little
creativity you might be able to parse the table data from the text
output of a given paper.

## Jeroen Ooms joins team rOpenSci!

A message from the team: We are happy to announce that [Jeroen
Ooms](https://jeroen.github.io/) has joined the rOpenSci crew! Jeroen is
a prolific programmer and author of numerous [widely used
packages](https://cran.r-project.org/web/checks/check_results_jeroen.ooms_at_stat.ucla.edu.html).
At rOpenSci, he will continue to work on developing awesome packages and
infrastructural software for improving the scientific tool chain.


Scientific articles are typically locked away in PDF format, a format designed primarily for printing but not so great for searching or indexing. The new pdftools package allows for extracting text and metadata from pdf files in R. From the extracted plain-text one could find articles discussing a particular drug or species name, without having to rely on publishers providing metadata, or pay-walled search engines.

Introducing pdftools - A fast and portable PDF extractor

David Winter

A new version of `rentrez`, our package for the NCBI's EUtils API, is
makingit's way around the CRAN mirrors. This release represents a
substantialimprovement to `rentrez`, including a [new
vignette](https://cran.r-project.org/web/packages/rentrez/vignettes/rentrez_tutorial.html)that
documents the whole package.

This posts describes some of the new things in `rentrez`, and gives us a
chanceto thank some of the people that have contributed to this
package's development.

## Thanks

Thanks to everyone who has filed and issue or written us an email about
`rentrez`,your contributions have been an important part of the
package's development. In particular, we welcome[Han
Guangchun](http://ewrebio.me) as a new contributor to `rentrez` and
thank[Matthew O'Meara](http://docking.org/~momeara/) for postingan issue
that brought the `by_id` mode for `entrez_link` (discussed below) to
ourattention.

\##The New Stuff

\###Extract elements from the results of `entrez_summary()`

The NCBI's "summary records" are very useful -- they provide the most
importantinformation about a given record in a relatively small and
simple file. `rentrez`provides the function `entrez_summary()` to
retrieve these records. When morethan one unique ID is passed to
`entrez_summary` the function returns a list ofesummary objects. For
instance, you could find all the genetic variants associatedwith asthma
by finding links between the [OMIM record for
asthma](https://www.omim.org/entry/600807) and records in the database
dbSNP:

::: {.highlight}
``` {.chroma tabindex="0"}
snps <- entrez_link(dbfrom="omim", db="snp", id= 600807)snp_summs <- entrez_summary(db="snp", id=snps$links$omim_snp)
```
:::

A very common use-case for `entrez_summary()` is to extract a subset of
theelements from each record in that list. This release includes the
function`extract_from_esummary` to make this as straightforward as
possible. It works witha single element to extract:

::: {.highlight}
``` {.chroma tabindex="0"}
extract_from_esummary(snp_summs, "chr")
```
:::

``` {tabindex="0"}
## 11079657  2786098  1031772  1031771   545659##     "17"      "1"      "2"      "2"     "11"
```

Or with multiple elements

::: {.highlight}
``` {.chroma tabindex="0"}
summary_table <- extract_from_esummary(snp_summs, c("chr", "global_maf", "fxn_class"))t(summary_table)
```
:::

``` {tabindex="0"}
##          chr  global_maf      fxn_class## 11079657 "17" "A=0.4295/2151" "intron-variant"## 2786098  "1"  "T=0.1569/786"  "intron-variant"## 1031772  "2"  "G=0.2131/1067" "downstream-variant-500B"## 1031771  "2"  "T=0.2582/1293" ""## 545659   "11" "C=0.3419/1712" "utr-variant-3-prime"
```

\###`entrez_link` can find external links

In addition to discovering links between records in NCBI databases, the
function`entrez_link` now provides support for finding external links
('linkouts' inNCBI terminology). Perhaps the most interesting example is
finding links for thefull text of articles in PubMed.

Let's try and find the full text of the paper describing
[taxize](https://github.com/ropensci/taxize) (using that article's
PMID). Tooverride the functions default behaviour (finding links within
NCBI databases)we set the `cmd` argument to `llinks` (short for library
links):

::: {.highlight}
``` {.chroma tabindex="0"}
taxize_links <- entrez_link(dbfrom="pubmed", id= 24555091, cmd="llinks")taxize_links
```
:::

``` {tabindex="0"}
## elink object with contents:##  $linkouts: links to external websites
```

The print function for this object tells you were the links live.

::: {.highlight}
``` {.chroma tabindex="0"}
taxize_links$linkouts
```
:::

``` {tabindex="0"}
## $ID_24555091## $ID_24555091[[1]]## Linkout from F1000 Research Ltd##  $Url: http://f1000research.com/a ...#### $ID_24555091[[2]]## Linkout from Europe PubMed Central##  $Url: http://europepmc.org/abstr ...#### $ID_24555091[[3]]## Linkout from PubMed Central##  $Url: http://www.ncbi.nlm.nih.go ...#### $ID_24555091[[4]]## Linkout from PubMed Central Canada##  $Url: http://pubmedcentralcanada ...
```

Each of those elements has a lot of information, but the URLs for each
objectare probably the most important. For this reason, `rentrez`
provides a functionto get just the URLs:

::: {.highlight}
``` {.chroma tabindex="0"}
linkout_urls(taxize_links)
```
:::

``` {tabindex="0"}
## $ID_24555091## [1] "http://f1000research.com/articles/10.12688/f1000research.2-191.v2/doi"## [2] "http://europepmc.org/abstract/MED/24555091"## [3] "http://www.ncbi.nlm.nih.gov/pmc/articles/pmid/24555091/"## [4] "http://pubmedcentralcanada.ca/pmcc/articles/pmid/24555091"
```

\###Web History features are easier to use

The NCBI provides a "Web History" feature to let users store the results
of theirsearches on the NCBI's severs and refer to those results without
having topass unique ID's back and forth between computers. These
features have always beensupported by `rentrez` but this release makes
them easier to use.

Specifically, when the new optional argument `use_history` is set to
`TRUE`functions will return a `web_history` object which can be used in
the place of uniqueIDs in calls to `entrez_fetch`, `entrez_summary` or
`entrez_link`.

To demonstrate, let's search for PubMed articles about the ciliate
genus*Tetrahymena*:

::: {.highlight}
``` {.chroma tabindex="0"}
Tet_papers <- entrez_search(db="pubmed", term="Tetrahymena[ORGN]", use_history=TRUE)Tet_papers
```
:::

``` {tabindex="0"}
## Entrez search result with 6599 hits (object contains 20 IDs and a web_history object)##  Search term (as translated):  "tetrahymena"[MeSH Terms] OR "tetrahymena"[All Fie ...
```

Now that we have a web_history object, we can use that to retrieve XML
representationsof the first 10 records:

::: {.highlight}
``` {.chroma tabindex="0"}
recs <- entrez_fetch(db="pubmed",                     web_history=Tet_papers$web_history,                     retmax=10, rettype="xml")
```
:::

\###It's easier to keep track of which records are linked to other
records

By default, when `entrez_link` gets a vector of more than one unique ID,
itreturns sets of linked-IDs that match *any* of the IDs in the original
call.That means the user loses track of the mapping between the original
IDs and thosefrom the linked database.

As of this release, `rentrez` supports the NCBI's `by_id` mode, which
solves this problem.Setting the new argument `by_id` to `TRUE` returns a
list, with each element ofthat list containing links for only one ID. To
demonstrate, let's find proteinsequences associated with specific genes
in the NCBI `gene` database:

::: {.highlight}
``` {.chroma tabindex="0"}
all_links  <- entrez_link(db="protein",                          dbfrom="gene",                          id=c(93100, 223646),                          by_id=TRUE)all_links
```
:::

``` {tabindex="0"}
## List of 2 elink objects,each containing##   $links: IDs for linked records from NCBI##
```

As you can see, printing the returned object let's you know what each
elementcontains, and you can extract the specific links you are looking
for easily:

::: {.highlight}
``` {.chroma tabindex="0"}
lapply(all_links, function(x) x$links$gene_protein)
```
:::

``` {tabindex="0"}
## [[1]]##  [1] "768043930" "767953815" "558472750" "194394158" "166221824"##  [6] "154936864" "119602646" "119602645" "119602644" "119602643"## [11] "119602642" "37787309"  "37787307"  "37787305"  "33991172"## [16] "21619615"  "10834676"#### [[2]]##  [1] "148697547" "148697546" "81899807"  "74215266"  "74186774"##  [6] "37787317"  "37589273"  "31982089"  "26339824"  "26329351"
```

## The rest

There are also numerous small changes that improve `rentrez`, fix bugs
andextend the package's documentation. We hope you find this new release
helpful,and as always we welcome bug reports via the [package's github
repository](https://github.com/ropensci/rentrez/issues).


A new version of rentrez, our package for the NCBI’s EUtils API, is makingit’s way around the CRAN mirrors. This release represents a substantialimprovement to rentrez, including a new vignettethat documents the whole package.

Rentrez 1_0 released

Carl Boettiger

We're happy to announce the launch of a CRAN-style repository for
rOpenSci at <http://packages.ropensci.org>

This repository contains the latest nightly builds from the master
branch of all rOpenSci packages currently on GitHub. This allows users
to install development versions of our software without specialized
functions such as `install_github()`, allowsdependencies not hosted on
CRAN to still be resolved automatically, and permits the use of
`update.packages()`.

## Using the repository

To use, simply add`packages.ropensci.org` to your existing list of R
repos, such as:

::: {.highlight}
``` {.chroma tabindex="0"}
options(repos = c("http://packages.ropensci.org", getOption("repos"))
```
:::

(If you don't have any default CRAN mirrors selected yet by
`getOption("repos")`, you may want to add one now). You can also include
this line in specific `install.packages()` requests:

::: {.highlight}
``` {.chroma tabindex="0"}
install.packages("taxize", repos = c("http://packages.ropensci.org", "http://cran.rstudio.com"))
```
:::

## Design

Our goal in creating a CRAN-style package repository (yes, it's
confusing that we use the word "repository" to describe both an
individual package source in a GitHub *repo* as well as a collection of
package binaries on a CRAN-like *repo*... sorry) was to provide users
with a way to install the latest development versions of rOpenSci
packages that offered an easier and more seamless alternative to the
widely used method of `devtools::install_github()`. This would be
particularly useful for updating all packages at once, or installing
development versions that depended on other versions of packages not yet
released to CRAN. As an added benefit, we also wanted a system that
would allow us to compute anonymized download statistics, analogous to
what RStudio provides for it's CRAN mirror. Getting this all to work
required the introduction of a few additional technologies.

### `drat` and `drat.builder`

While the basic structure of a CRAN-like R repository is simple, and
used by both platforms such as [RForge](https://r-forge.r-project.org/)
and individual developers, kudos really goes to Dirk Eddelbuettel's
[drat](https://github.com/eddelbuettel/drat) package for really making
this automated, simple, and fun. While `drat` makes it easy to toss
individual packages into a CRAN-like repo (which we often refer to as a
`drat` repo), we needed an easy & automatic way to add a whole list of
packages, given their GitHub repos. Rich FitzJohn's new
[drat.builder](https://github.com/richfitz/drat.builder) package does
precisely this; handling the downloading of packages with some clever
record-keeping to avoid building and adding packages which have not
changed since the last time the `drat` repo was built. The sources for
building the rOpenSci packages repository can be found in our "drat"
GitHub repo: <https://github.com/ropensci/drat>

### Dynamic package lists: `ropkgs`

With rOpenSci, we wanted to take this one step further. No one wants to
have to maintain one more list that must be updated every time a package
is successfully on-boarded to the project. Scott Chamberlain's work with
[ropkgs](https://github.com/ropensci/ropkgs) provides a convenient way
to query the rOpenSci software suite, automatically generating a list of
available rOpenSci packages, and filtering them on relevant metadata,
such as those that are in good status and installable condition, like
so:

::: {.highlight}
``` {.chroma tabindex="0"}
library("ropkgs")out <- ro_pkgs()good <- out$packages$status == "good"installable <- out$packages$installablepkgs <- out$packages$name[installable & good]
```
:::

### The magic of continuous integration: CircleCI

With `ropkgs`, `drat` and `drat.builder`, we now have everything we need
to automate the building of the CRAN-like package repository. Now we
just need some computing resource that can do the hard work of pulling
down all the GitHub packages, building the repository, and securely
sending off the binaries somewhere they can be downloaded. Continuous
Integration systems turn out to be perfect for this. My favorite CI
platform at the moment is [CircleCi](https://circleci.com), for several
reasons particularly relevant here:

- it has a rich API which includes support for `POST` requests which can
  trigger a build without making commits to GitHub
- it supports custom Docker containers, allowing us to just download a
  container with most or all the dependencies we need to build packages,
  etc., without having to wait for them to install manually from source
  first.
- it has a convenient web interface for providing secure credentials
  we'll need to publish the binary repository to GitHub or Amazon.

Circle has other advantages too, like great live help and the ability to
ssh into your CI run to troubleshoot when all else fails, but otherwise
works like most other CI platforms. More on that another day. The daily
builds are triggered by a simple `POST` request running as a cron job.
The circle.yml configuration file appears in the project's drat repo --
check out how simple it is!

### Publishing to Amazon

The last step would be getting download logs; which is somewhat more
complicated than it sounds. `drat` conveniently already handles pushing
packages to GitHub's gh-pages, a free and easy way to provide static
hosting. This is free and easy, but isn't ideal, particularly for large
and frequently updated package collections. Also, it is impossible to
get download logs from this approach. To avoid these issues, we settled
on pushing our package repository to a static site hosted through
Amazon's S3 data storage "buckets." It's not free, but for at most a few
gigs of space we'll need it's still very cost effective. In particular,
S3 buckets can generate their own log files, which provides a way to
count package downloads.

Secure communication with Amazon S3 system is accomplished using the
very nascent / actively developing
[aws.s3](https://github.com/cloudyr/aws.s3) R package from the awesome
[cloudyr](https://cloudyr.github.io/) project.

### Parsing the download logs

Amazon's S3 logs are rather raw and require some good ol data tidying
work to transform them into the conveniently parsed, tidied and
IP-address-anonymized format used by RStudio's download logs. Eventually
this too can be accomplished by the CircleCI builds, but at the moment
is too computationally intensive for them. A script for this work-flow
can be found in the project repo,
[parse_s3_logs.R](https://github.com/ropensci/drat/blob/gh-pages/parse_s3_logs.R).
As the data accumulate we should be able to start publishing the tidy
logs.

This project is still in it's early days, and as ever, we welcome
feedback, problems or ideas on the [issues
tracker](https://github.com/ropensci/drat/issues).

------------------------------------------------------------------------

Now go ahead and install or update some packages from the shiny new
<http://packages.ropensci.org>!


We’re happy to announce the launch of a CRAN-style repository for rOpenSci at http://packages.ropensci.org This repository contains the latest nightly builds from the master branch of all rOpenSci packages currently on GitHub.

Rogue Scholar Beiträge

Introducing pdftools - A fast and portable PDF extractor

Rentrez 1_0 released

A drat repository for rOpenSci