University of Glasgow

Roderic Page

::: {.separator style="clear: both;"}
[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBaMrDj0KRPFPHmjSBYzffLbJZhA711GGekWPbiRqzSpSYLvBdef7OLwDrPaarFrxMHrBEQP5iiM_u3hF_AdNODs5whiO0i5cJmoQx3NWDcWUroU9OGF9o99BMcOIkIoCTETHH/s200/Rf7UoXTw_400x400.jpg){border="0"
width="128" original-height="240"
original-width="240"}](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBaMrDj0KRPFPHmjSBYzffLbJZhA711GGekWPbiRqzSpSYLvBdef7OLwDrPaarFrxMHrBEQP5iiM_u3hF_AdNODs5whiO0i5cJmoQx3NWDcWUroU9OGF9o99BMcOIkIoCTETHH/s240/Rf7UoXTw_400x400.jpg){style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"}
:::

The [Plazi](http://plazi.org) project has become one of the major
contributors to GBIF with some 36,000 datasets yielding some 500,000
occurrences (see [Plazi\'s GBIF
page](https://www.gbif.org/publisher/7ce8aef0-9e92-11dc-8738-b8a03c50a862)
for details). These occurrences are extracted from taxonomic publication
using automated methods. New data is published almost daily (see [latest
treatments](https://tb.plazi.org/GgServer/static/newToday.html)). The
map below shows the geographic distribution of material citations
provided to GBIF by Plazi, which gives you a sense of the size of the
dataset.

::: {.separator style="clear: both;"}
[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbzyO3NVCA0r2-2bkxUBCZdA9iuZZATf_mlqYDy6CCZjO4NTSEQMC8txdCLGGo_Ynz7ISTEGVLuPJDX3Rx8ZOh1vDZVzseKKa6OsvPsc2qoirL_tGwF4i0JJEWR5Uhyphenhyphen8b1R4Ey/s400/Screenshot+2021-10-24+at+20.43.23.png){border="0"
width="400" original-height="514"
original-width="1030"}](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbzyO3NVCA0r2-2bkxUBCZdA9iuZZATf_mlqYDy6CCZjO4NTSEQMC8txdCLGGo_Ynz7ISTEGVLuPJDX3Rx8ZOh1vDZVzseKKa6OsvPsc2qoirL_tGwF4i0JJEWR5Uhyphenhyphen8b1R4Ey/s1030/Screenshot+2021-10-24+at+20.43.23.png){style="display: block; padding: 1em 0; text-align: center; "}
:::

By any metric Plazi represents a considerable achievement. But often
when I browse individual records on Plazi I find records that seem
clearly incorrect. Text mining the literature is a challenging problem,
but at the moment Plazi seems something of a \"black box\". PDFs go in,
the content is mined, and data comes up to be displayed on the Plazi web
site and uploaded to GBIF. Nowhere does there seem to be an evaluation
of how accurate this text mining actually is. Anecdotally it seems to
work well in some cases, but in others it produces what can only be
described as bogus records.

## Finding errors

A treatment in Plazi is a block of text (and sometimes illustrations)
that refers to a single taxon. Often that text will include a
description of the taxon, and list one or more specimens that have been
examined. These lists of specimens (\"material citations\") are one of
the key bits of information that Plaza extracts from a treatment as
these citations get fed into GBIF as occurrences.

To help explore treatments I\'ve constructed a simple web site that
takes the Plazi identifier for a treatment and displays that treatment
with the material citations highlighted. For example, for the Plazi
treatment
[03B5A943FFBB6F02FE27EC94FABEEAE7](https://tb.plazi.org/GgServer/html/03B5A943FFBB6F02FE27EC94FABEEAE7)
you can view the marked up version at
<https://plazi-tester.herokuapp.com/?uri=622F7788-F0A4-449D-814A-5B49CD20B228>.
Below is an example of a material citation with its component parts
tagged:

::: {.separator style="clear: both;"}
[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjinptnav7_CNmsrnCzCArizBz19_srW3AyQ8LL5b5PHqFw-m3zjXDDqxd4PMCacnTQ2xyJh61Wo_0omEbwQ5AGJRsHteeqa7-btTty4QGcVsdTVp0CpOxZrw7hFkpwBGWN_lea/s400/Screenshot+2021-10-24+at+20.59.56.png){border="0"
width="400" original-height="94"
original-width="693"}](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjinptnav7_CNmsrnCzCArizBz19_srW3AyQ8LL5b5PHqFw-m3zjXDDqxd4PMCacnTQ2xyJh61Wo_0omEbwQ5AGJRsHteeqa7-btTty4QGcVsdTVp0CpOxZrw7hFkpwBGWN_lea/s693/Screenshot+2021-10-24+at+20.59.56.png){style="display: block; padding: 1em 0; text-align: center; "}
:::

This is an example where Plazi has successfully parsed the specimen. But
I keep coming across cases where specimens have not been parsed
correctly, resulting in issues such as single specimens being split into
multiple records (e.g.,
<https://plazi-tester.herokuapp.com/?uri=5244B05EFFC8E20F7BC32056C178F496>),
geographical coordinates being misinterpreted (e.g.,
<https://plazi-tester.herokuapp.com/?uri=0D228E6AFFC2FFEFFF4DE8118C4EE6B9>),
or collector\'s initials being confused with codes for natural history
collections (e.g.,
<https://plazi-tester.herokuapp.com/?uri=252C87918B362C05FF20F8C5BFCB3D4E>).

Parsing specimens is a hard problem so it\'s not unexpected to find
errors. But they do seem common enough to be easily found, which raises
the question of just what percentage of these material citations are
correct? How much of the data Plazi feeds to GBIF is correct? How would
we know?

## Systemic problems

Some of the errors I\'ve found concern the interpretation of the parsed
data. For example, it is striking that despite including marine taxa
**no** Plazi record has a value for depth below sea level (see [GBIF
search on depth range 0-9999 for
Plazi](https://www.gbif.org/occurrence/map?depth=0,9999&publishing_org=7ce8aef0-9e92-11dc-8738-b8a03c50a862&advanced=1)).
But [many records do have an
elevation](https://www.gbif.org/occurrence/map?elevation=0,9999&publishing_org=7ce8aef0-9e92-11dc-8738-b8a03c50a862&advanced=1),
including records from marine environments. Any record that has a depth
value is interpreted by Plazi as being elevation, so we have aerial
crustacea and fish.

### Map of Plazi records with depth 0-9999m

::: {.separator style="clear: both;"}
[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnYHyMsNeElQaa-WjfVe6PHQfPx5O4sbAt6Spxn2Cp8FPPK50JbuKCg0o188uAg8ryVM_aZtYvBSl1tpAdAn25yIvmSh97wf1fxa6nb4C9ijNy_hOTtvK3k1aA3HbWJ6xpSma2/s320/Screenshot+2021-10-25+at+12.03.51.png){border="0"
width="400" original-height="258"
original-width="673"}](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnYHyMsNeElQaa-WjfVe6PHQfPx5O4sbAt6Spxn2Cp8FPPK50JbuKCg0o188uAg8ryVM_aZtYvBSl1tpAdAn25yIvmSh97wf1fxa6nb4C9ijNy_hOTtvK3k1aA3HbWJ6xpSma2/s673/Screenshot+2021-10-25+at+12.03.51.png){style="display: block; padding: 1em 0; text-align: center; "}
:::

### Map of Plazi records with elevation 0-9999m

::: {.separator style="clear: both;"}
[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4kucSwi2YxsVsjDboeeZM6Nyy9W0REaqnXEsqgdk2tn_oqzMflKMzqpWnXJCwXvpKD1EOhhtN4CA9B20dJFcMycFpQQJGSlRD-BaWQMv6hApysjFWHkReaaeqYnvwrG0-yjQi/s320/Screenshot+2021-10-25+at+12.04.06.png){border="0"
width="400" original-height="256"
original-width="675"}](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4kucSwi2YxsVsjDboeeZM6Nyy9W0REaqnXEsqgdk2tn_oqzMflKMzqpWnXJCwXvpKD1EOhhtN4CA9B20dJFcMycFpQQJGSlRD-BaWQMv6hApysjFWHkReaaeqYnvwrG0-yjQi/s675/Screenshot+2021-10-25+at+12.04.06.png){style="display: block; padding: 1em 0; text-align: center; "}
:::

Anecdotally I\'ve also noticed that Plazi seems to do well on zoological
data, especially journals like *Zootaxa*, but it often struggles with
botanical specimens. Botanists tend to cite specimens rather differently
to zoologists (botanists emphasise collector numbers rather than
specimen codes). Hence data quality in Plazi is likely to taxonomic
biased.

Plazi is [using GitHub to track issues with
treatments](https://github.com/plazi/community/issues) so feedback on
erroneous records is possible, but this seems inadequate to the task.
There are tens of thousands of data sets, with more being released
daily, and hundreds of thousands of occurrences, and relying on GitHub
issues devolves the responsibility for error checking onto the data
users. I don\'t have a measure of how many records in Plazi have
problems, but because I suspect it is a significant fraction because for
any given day\'s output I can typically find errors.

## What to do?

Faced with a process that generates noisy data there are several of
things we could do:

1.  Have tools to detect and flag errors made in generating the data.
2.  Have the data generator give estimates the confidence of its
    results.
3.  Improve the data generator.

I think a comparison with the problem of parsing bibliographic
references might be instructive here. There is a long history of people
developing tools to parse references ([I\'ve even had a
go](https://iphylo.blogspot.com/2021/05/finding-citations-of-specimens.html)).
State-of-the art tools such as [AnyStyle](https://anystyle.io) feature
machine learning, and are tested against [human curated
datasets](https://github.com/inukshuk/anystyle/blob/master/res/parser/core.xml)
of tagged bibliographic records. This means we can evaluate the
performance of a method (how well does it retrieve the same results as
human experts?) and also improve the method by expanding the corpus of
training data. Some of these tools can provide a measures of how
confident they are when classifying a string as, say, a person\'s name,
which means we could flag potential issues for anyone wanting to use
that record.

We don\'t have equivalent tools for parsing specimens in the literature,
and hence have no easy way to quantify how good existing methods are,
nor do we have a public corpus of material citations that we can use as
training data. I [blogged about
this](https://iphylo.blogspot.com/2021/05/finding-citations-of-specimens.html)
a few months ago and was considering using Plazi as a source of marked
up specimen data to use for training. However based on what I\'ve looked
at so far Plazi\'s data would need to be carefully scrutinised before it
could be used as training data.

Going forward, I think it would be desirable to have a set of records
that can be used to benchmark specimen parsers, and ideally have the
parsers themselves available as web services so that anyone can evaluate
them. Even better would be a way to contribute to the training data so
that these tools improve over time.

Plazi\'s data extraction tools are mostly desktop-based, that is, you
need to download software to use their methods. However, there are
experimental web services available as well. I\'ve created a simple
wrapper around the material citation parser, you can try it at
<https://plazi-tester.herokuapp.com/parser.php>. It takes a single
material citation and returns a version with elements such as specimen
code and collector name tagged in different colours.

## Summary

Text mining the taxonomic literature is clearly a gold mine of data, but
at the same time it is potentially fraught as we try and extract
structured data from semi-structured text. Plazi has demonstrated that
it is possible to extract a lot of data from the literature, but at the
same time the quality of that data seems highly variable. Even minor
issues in parsing text can have big implications for data quality (e.g.,
marine organisms apparently living above sea level). Historically in
biodiversity informatics we have favoured data quantity over data
quality. Quantity has an obvious metric, and has milestones we can
celebrate (e.g., [one billion
specimens](GBIF%20at%201%20billion%20-%20what's%20next?)). There aren\'t
really any equivalent metrics for data quality.

Adding new types of data can sometimes initially result in a new set of
quality issues (e.g., [GBIF metagenomics and
metacrap](https://iphylo.blogspot.com/2019/12/gbif-metagenomics-and-metacrap.html))
that take time to resolve. In the case of Plazi, I think it would be
worthwhile to quantify just how many records have errors, and develop
benchmarks that we can use to test methods for extracting specimen data
from text. If we don\'t do this then there will remain uncertainty as to
how much trust we can place in data mined from the taxonomic literature.

## Update

Plazi has responded, see [Liberating material citations as a first step
to more better
data](http://plazi.org/posts/2021/10/liberation-first-step-toward-quality/).
My reading of their repsonse is that it essentially just reiterates
Plazi\'s approach and doesn\'t tackle the underlying issue: their method
for extracting material citations is error prone, and many of those
errors end up in GBIF.


The Plazi project has become one of the major contributors to GBIF with some 36,000 datasets yielding some 500,000 occurrences (see Plazi's GBIF page for details). These occurrences are extracted from taxonomic publication using automated methods.

Problems with Plazi parsing: how reliable are automated methods for extracting specimens from the literature?

Quick note on a tool I\'ve been working on to parse citations, that is
to take a series of strings such as:

- Möllendorff O (1894) On a collection of land-shells from the Samui
  Islands, Gulf of Siam. Proceedings of the Zoological Society of
  London, 1894: 146--156.
- de Morgan J (1885) Mollusques terrestres & fluviatiles du royaume de
  Pérak et des pays voisins (Presqúile Malaise). Bulletin de la Société
  Zoologique de France, 10: 353--249.
- Morlet L (1889) Catalogue des coquilles recueillies, par M. Pavie dans
  le Cambodge et le Royaume de Siam, et description ďespèces nouvelles
  (1). Journal de Conchyliologie, 37: 121--199.
- Naggs F (1997) William Benson and the early study of land snails in
  British India and Ceylon. Archives of Natural History, 24:37--88.

and return structured data. This is an old problem, and pretty much a
\"solved\" problem. See for example [AnyStyle](https://anystyle.io).
I\'ve played with AnyStyle and it\'s great, but I had to install it on
my computer rather than simply use it as a web service. I also wanted to
explore the approach a bit more as a possible a model for [finding
citations of
specimens](https://iphylo.blogspot.com/2021/05/finding-citations-of-specimens.html).

> Loving Sylvester Keil\'s AnyStyle reference parser
> <https://t.co/pcbvctr5vf>, but not loving the whole Ruby experience
> (whadda mean I need to upgrade Ruby on my Mac to install it?). Oh for
> a Docker version of a web service\... still, very cool tool.
>
> --- Roderic Page (@rdmpage) [July 7,
> 2020](https://twitter.com/rdmpage/status/1280471053660192768?ref_src=twsrc%5Etfw)

After trying to install the underlying conditional random fields (CRF)
engine used by AnyStyle and running into a bunch of errors, I switched
to a tool I could get working, namely CRF++. After figuring out how to
[compiling a C++ application to run on
Heroku](https://iphylo.blogspot.com/2021/06/compiling-c-application-to-run-on-heroku.html)
I started to wonder how to use this as the basis of a citation parser.
Fortunately, I had used the Perl-based
[ParsCit](https://github.com/knmnyn/ParsCit) years ago, and managed to
convert the relevant bits to PHP and build a simple web service around
it.

Although I\'ve abandoned the Ruby-based AnyStyle I do use AnyStyle\'s
XML format for the training data. I also built a crude editor to create
small training data sets that uses a technique published by the author
of the blogging tool I\'m using to write this post (see [MarsEdit Live
Source
Preview](https://red-sweater.com/blog/3025/marsedit-live-source-preview)).
Typically I use this to correctly annotate examples where the parser
failed. Over time I add these to the training data and the performance
gets better.

This is pretty much a side project or a side project, but ultimately the
goal is to employ it to help extract citation data from publications,
both to generate data to populate ([BioStor](https://biostor.org)), and
also start to flesh out the citation graph for publications in Wikidata.

If you want to play with the tool it is at
<https://citation-parser.herokuapp.com>. At the moment it takes some
citation strings and returns the result in CSL-JSON, which is becoming
the default way to represent structured bibliographic data. Code is on
[GitHub](https://github.com/rdmpage/citation-parsing).


Quick note on a tool I've been working on to parse citations, that is to take a series of strings such as: Möllendorff O (1894) On a collection of land-shells from the Samui Islands, Gulf of Siam. Proceedings of the Zoological Society of London, 1894: 146–156. de Morgan J (1885) Mollusques terrestres &amp; fluviatiles du royaume de Pérak et des pays voisins (Presqúile Malaise). Bulletin de la Société Zoologique de France, 10: 353–249.

Citation parsing tool released

Scott Chamberlain

[pubchunks](https://github.com/ropensci/pubchunks) is a package grown
out of the [fulltext](https://github.com/ropensci/fulltext) package.
`fulltext`provides a single interface to many sources of full text
scholarly articles. Aspart of the user flow in `fulltext` there is an
extraction step where `fulltext::chunks()`pulls parts of articles out of
XML format article files.

As part of making `fulltext` more maintainable and focused on simply
fetching articles,and realizing that pulling out bits of structured XML
files is a more general problem,we broke out `pubchunks` into a separate
package. `fulltext::ft_chunks()` and`fulltext::ft_tabularize()` will
eventually be removed and we'll point users to`pubchunks`.

The goal of `pubchunks` is to fetch sections out of XML format scholarly
articles.Users do not need to know about XML and all of its warts. They
only need to knowwhere their files or XML strings are and what sections
they want ofeach article. Then the user can combine these sections and
do whatever they wishdownstream; for example, analysis of the text
structure or a meta-analysis combiningp-values or other data.

The other major format, and more common format, that articles come in is
PDF. However,PDF has no structure other than perhaps separate pages, so
it's not really possibleto easily extract specific sections of an
article. Some publishers provide absolutelyno XML versions (cough,
Wiley) while others that do a good job of this are almostentirely
paywalled (cough, Elsevier). There are some open access publishers
thatdo provide XML (PLOS, Pensoft, Hindawi) - so you have the best of
both worlds withthose publishers.

`pubchunks` is still in early days of development, so we'd love any
feedback.

- pubchunks on GitHub: <https://github.com/ropensci/pubchunks>
- pubchunks on CRAN: <https://cran.rstudio.com/web/packages/pubchunks/>

## Functions in pubchunks

All exported functions are prefixed with `pub` to help reduce namespace
conflicts.

The two main functions are:

- `pub_chunks()`: fetch XML sections
- `pub_tabularize()`: coerce output of `pub_chunks()` into data.frame's

Other functions that you may run in to:

- `pub_guess_publisher()`: guess publisher from XML file or character
  string
- `pub_sections()`: sections `pubchunks` knows how to handle
- `pub_providers()`: providers (i.e., publishers) `pubchunks` knows how
  to handle explicitly

## How it works

When using `pub_chunks()` we first figure out what publisher the XML
comes from. We do thisbeacause journals from the same publisher
often/usually follow the same format/structure, sowe can be relatively
confident of rules for pulling out certain sections.

Once we have the publisher, we go through each section of the article
the user reqeusts, anduse the publisher specific
[XPATH](https://en.wikipedia.org/wiki/XPath) (an XML query language)that
we've written to extract that section from the XML.

The ouput is a named list, where names are the sections; and the output
is an S3 class witha print method to make it more easily digestable.

## Installation

The first version of the package has just hit CRAN, so not all binaries
are available as of thisdate.

::: {.highlight}
``` {.chroma tabindex="0"}
install.packages("pubchunks")
```
:::

You may have to install the dev version

::: {.highlight}
``` {.chroma tabindex="0"}
remotes::install_github("ropensci/pubchunks")
```
:::

After sending to CRAN a few days back I noticed a number of things that
needed fixing/improving,so you can also try out those fixes:

::: {.highlight}
``` {.chroma tabindex="0"}
remotes::install_github("ropensci/pubchunks@fix-pub_chunks")
```
:::

Load the package

::: {.highlight}
``` {.chroma tabindex="0"}
library(pubchunks)
```
:::

## Pub chunks

`pub_chunks` accepts a file path, XML as character string,
`xml_document` object, or a list ofthose three things.

We currently support the following publishers, though not all sections
below are allowedin every publisher:

- elife
- plos
- elsevier
- hindawi
- pensoft
- peerj
- copernicus
- frontiers
- f1000research

We currently support the following sections, not all of which are
supported, or make sense, foreach publisher:

- front - Publisher, journal and article metadata elements
- body - Body of the article
- back - Back of the article, acknowledgments, author contributions,
  references
- title - Article title
- doi - Article DOI
- categories - Publisher's categories, if any
- authors - Authors
- aff - Affiliation (includes author names)
- keywords - Keywords
- abstract - Article abstract
- executive_summary - Article executive summary
- refs - References
- refs_dois - References DOIs - if available
- publisher - Publisher name
- journal_meta - Journal metadata
- article_meta - Article metadata
- acknowledgments - Acknowledgments
- permissions - Article permissions
- history - Dates, recieved, published, accepted, etc.

Get an example XML file from the package

::: {.highlight}
``` {.chroma tabindex="0"}
x <- system.file("examples/pensoft_1.xml", package = "pubchunks")
```
:::

Pass to `pub_chunks` and state which section(s) you want

::: {.highlight}
``` {.chroma tabindex="0"}
pub_chunks(x, sections = "abstract")#> <pub chunks>#>   from: character#>   sections: abstract#>    abstract (n=1): AbstractNineteen species of seed-beetles belonging ...pub_chunks(x, sections = "aff")#> <pub chunks>#>   from: character#>   sections: aff#>    aff (n=7): nested listpub_chunks(x, sections = "title")#> <pub chunks>#>   from: character#>   sections: title#>    title (n=1): Contribution to the knowledge of seed-beetles (Col ...pub_chunks(x, sections = c("abstract", "title", "authors", "refs"))#> <pub chunks>#>   from: character#>   sections: abstract, title, authors, refs#>    abstract (n=1): AbstractNineteen species of seed-beetles belonging ...#>    title (n=1): Contribution to the knowledge of seed-beetles (Col ...#>    authors (n=7): nested list#>    refs (n=13): AntonKW (2010) Catalogue of Palaearctic Coleoptera
```
:::

You can also pass in a character string of XML, e.g.,

::: {.highlight}
``` {.chroma tabindex="0"}
xml_str <- paste0(readLines(x), collapse = "\n")class(xml_str)#> [1] "character"pub_chunks(xml_str, sections = "title")#> <pub chunks>#>   from: character#>   sections: title#>    title (n=1): Contribution to the knowledge of seed-beetles (Col ...
```
:::

Or an `xml_document` object from `xml2::read_xml`

::: {.highlight}
``` {.chroma tabindex="0"}
xml_doc <- xml2::read_xml(x)class(xml_doc)#> [1] "xml_document" "xml_node"pub_chunks(xml_doc, sections = "title")#> <pub chunks>#>   from: xml_document#>   sections: title#>    title (n=1): Contribution to the knowledge of seed-beetles (Col ...
```
:::

Or pass in a list of the above objects, e.g., here a list of file paths

::: {.highlight}
``` {.chroma tabindex="0"}
pensoft_xml <- system.file("examples/pensoft_1.xml", package = "pubchunks")peerj_xml <- system.file("examples/peerj_1.xml", package = "pubchunks")copernicus_xml <- system.file("examples/copernicus_1.xml", package = "pubchunks")frontiers_xml <- system.file("examples/frontiers_1.xml", package = "pubchunks")pub_chunks(  list(pensoft_xml, peerj_xml, copernicus_xml, frontiers_xml),  sections = c("abstract", "title", "authors", "refs"))#> [[1]]#> <pub chunks>#>   from: character#>   sections: abstract, title, authors, refs#>    abstract (n=1): AbstractNineteen species of seed-beetles belonging ...#>    title (n=1): Contribution to the knowledge of seed-beetles (Col ...#>    authors (n=7): nested list#>    refs (n=13): AntonKW (2010) Catalogue of Palaearctic Coleoptera#> #> [[2]]#> <pub chunks>#>   from: character#>   sections: abstract, title, authors, refs#>    abstract (n=1): Climate change is predicted to lead to more extrem ...#>    title (n=1): Storm effects on intertidal invertebrates: increas ...#>    authors (n=7): nested list#>    refs (n=60): Alcántara-Carrió et al. (2017)Alcántara-CarrióJSas#> #> [[3]]#> <pub chunks>#>   from: character#>   sections: abstract, title, authors, refs#>    abstract (n=1): Soil temperatures at various depths are unique par ...#>    title (n=1): Quality control of 10-min soil temperatures data a ...#>    authors (n=3): nested list#>    refs (n=9): 1Bertrand, C., Gonzalez Sotelino, L., and Journée,#> #> [[4]]#> <pub chunks>#>   from: character#>   sections: abstract, title, authors, refs#>    abstract (n=1): Our current understanding of Antarctic soils is de ...#>    title (n=1): Metagenomic Analysis of a Southern Maritime Antarc ...#>    authors (n=8): nested list#>    refs (n=56): AislabieJ.BroadyP.SaulD. (2006). Culturable hetero#> #> attr(,"ft_data")#> [1] FALSE
```
:::

Last, since we broke `pubchunks` out of `fulltext` package we support
`fulltext`here as well.

::: {.highlight}
``` {.chroma tabindex="0"}
library("fulltext")dois <- c('10.1371/journal.pone.0086169', '10.1371/journal.pone.0155491',   '10.7554/eLife.03032')x <- fulltext::ft_get(dois)pub_chunks(fulltext::ft_collect(x), sections="authors")#> $plos#> $plos$`10.1371/journal.pone.0086169`#> <pub chunks>#>   from: xml_document#>   sections: authors#>    authors (n=4): nested list#> #> $plos$`10.1371/journal.pone.0155491`#> <pub chunks>#>   from: xml_document#>   sections: authors#>    authors (n=9): nested list#> #> #> $elife#> $elife$`10.7554/eLife.03032`#> <pub chunks>#>   from: xml_document#>   sections: authors#>    authors (n=6): nested list#> #> #> attr(,"ft_data")#> [1] TRUE
```
:::

## Tabularize

It's great to pull out the sections you want, but most people will
likely want towork with data.frame's instead of lists. `pub_tabularize`
is the answer:

::: {.highlight}
``` {.chroma tabindex="0"}
x <- system.file("examples/elife_1.xml", package = "pubchunks")res <- pub_chunks(x, c("doi", "title", "keywords"))pub_tabularize(res)#>                   doi                                          title#> 1 10.7554/eLife.03032 MicroRNA-mediated repression of nonsense mRNAs#> 2 10.7554/eLife.03032 MicroRNA-mediated repression of nonsense mRNAs#> 3 10.7554/eLife.03032 MicroRNA-mediated repression of nonsense mRNAs#> 4 10.7554/eLife.03032 MicroRNA-mediated repression of nonsense mRNAs#> 5 10.7554/eLife.03032 MicroRNA-mediated repression of nonsense mRNAs#> 6 10.7554/eLife.03032 MicroRNA-mediated repression of nonsense mRNAs#>                       keywords .publisher#> 1                     microRNA      elife#> 2            nonsense mutation      elife#> 3 nonsense-mediated mRNA decay      elife#> 4                          APC      elife#> 5             intron retention      elife#> 6  premature termination codon      elife
```
:::

It handles many inputs as well:

::: {.highlight}
``` {.chroma tabindex="0"}
out <- pub_chunks(  list(pensoft_xml, peerj_xml, copernicus_xml, frontiers_xml),  sections = c("doi", "title", "keywords"))pub_tabularize(out)#> [[1]]#>                                                                                                     title#> 1 Contribution to the knowledge of seed-beetles (Coleoptera, Chrysomelidae, Bruchinae) in Xinjiang, China#>   .publisher#> 1    pensoft#> #> [[2]]#>                                                                                                title#> 1 Storm effects on intertidal invertebrates: increased beta diversity of few individuals and species#>   .publisher#> 1      peerj#> #> [[3]]#>                      doi#> 1 10.5194/asr-12-23-2015#>                                                     title .publisher#> 1 Quality control of 10-min soil temperatures data at RMI copernicus#> #> [[4]]#>                        doi#> 1 10.3389/fmicb.2012.00403#>                                                        title .publisher#> 1 Metagenomic Analysis of a Southern Maritime Antarctic Soil  frontiers
```
:::

The output of `pub_tabularize` is a list of data.frame's. You can easily
combine the outputwith e.g. `rbind` or `data.table::rbindlist` or
`dplyr::bind_rows`. Here's an example with`data.table::rbindlist`:

::: {.highlight}
``` {.chroma tabindex="0"}
data.table::rbindlist(pub_tabularize(out), fill = TRUE)#>                                                                                                      title#> 1: Contribution to the knowledge of seed-beetles (Coleoptera, Chrysomelidae, Bruchinae) in Xinjiang, China#> 2:      Storm effects on intertidal invertebrates: increased beta diversity of few individuals and species#> 3:                                                 Quality control of 10-min soil temperatures data at RMI#> 4:                                              Metagenomic Analysis of a Southern Maritime Antarctic Soil#>    .publisher                      doi#> 1:    pensoft                     <NA>#> 2:      peerj                     <NA>#> 3: copernicus   10.5194/asr-12-23-2015#> 4:  frontiers 10.3389/fmicb.2012.00403
```
:::

## TO DO

We'll be adding support for more publishers (including right now working
on Pubmed XML formatarticles), more article sections, making sure
default section extraction is smart as possible,and more.

Of course, if you know XPATH and don't mind doing it, you can do what
this package does yourself.However, you will have to write different
XPATH for different publishers/journals, so leveragingthis approach
still may save some time.


pubchunks is a package grown out of the fulltext package. fulltextprovides a single interface to many sources of full text scholarly articles. Aspart of the user flow in fulltext there is an extraction step where fulltext::chunks()pulls parts of articles out of XML format article files.

Postagens de Rogue Scholar

Problems with Plazi parsing: how reliable are automated methods for extracting specimens from the literature?

Citation parsing tool released

pubchunks: extract parts of scholarly XML articles