Matthias Grenié

Hugo Gruson

We've been following rOpenSci's work for a long time, and we use several
packages on a daily basis for our scientific projects, especially
[taxize](https://github.com/ropensci/taxize/) to clean species names,
[rredlist](https://github.com/ropensci/rredlist/) to extract species
IUCN statuses or \[treeio\](many probs with this post) to work with
phylogenetic trees.rOpensci is a perfect incarnation of a vibrant and
diverse community where people learn and develop new ideas, especially
regarding scientific packages.We've also noticed how much the thorough
review process improves the quality of the packages that join the
rOpenSci ecosystem.And while we were admiring the dynamics of rOpenSci
community, we started to wonder how we could contribute to this
ecosystem.And this is how we started our quest to find a project that
could fit rOpenSci goals while at the same time teach us new skills.

Open Access is the idea that scientific articles should be available to
everyone to favour scientific dissemination, as well as public
information. Open Access is rising in the scientific community with more
and more public funding agencies requiring funded projects to make their
article open access.But several very different models exist under this
umbrella term of "Open Access".One of them is the so-called ["green open
access"](https://en.wikipedia.org/wiki/Open_access#Green_OA), where the
articles are made publicly available by their authors via their
deposition in institutional or public repositories, such as
[bioRxiv](https://www.biorxiv.org) or
[HAL](https://hal.archives-ouvertes.fr/).Scientific journals have
different policies regarding green open access: some let you archive
various versions of the manuscript right after acceptance, while others
ask you to wait an embargo period or forbid entirely the archival of the
manuscript.To support green open access, scientists can elect to publish
their work in journals that authorize manuscript archival.And this is
where [SHERPA/RoMEO](https://sherpa.ac.uk/romeo/index.php) comes in
handy: it offers a publicly available database of open access policies
of scientific journals and lists the conditions under which manuscript
archival might be allowed.

The SHERPA/RoMEO database is available through an [Application
Programming
Interface](https://en.wikipedia.org/wiki/Application_Programming_Interface)
(API), which meant that we could build an R client to programmatically
access this data.This would allow researchers to more easily select
journals based on their manuscript archival policies. An R client would
also be a precious tool for bibliometricians who want to get the general
picture of open access practices in a particular subfield.

Although we had no prior experience working with web data or scraping
APIs for data, we knew several tools existed to interact with APIs in R
such as [httr](https://github.com/r-lib/httr) or rOpenSci's
[crul](https://github.com/ropensci/crul).And as we started the
development of [rromeo](https://github.com/ropensci/rromeo), we also
knew we could use the many examples of R client for APIs available on
the [CRAN Task View on Web
Technologies](https://cran.r-project.org/web/views/WebTechnologies.html)
as models.

### SHERPA/RoMEO API

SHERPA/RoMEO has been available through a web interface since at least
February 2004 according to the [Internet Archive](https://archive.org/)
and their API was released in December 2006 (from the Internet Archive
<https://web.archive.org/web/20061211125806/http://www.sherpa.ac.uk:80/romeo.php>),
making their data available to anyone since then.

There are three versions of the manuscript considered in SHERPA/RoMEO:

1.  **pre-print**, which is the manuscript version **before peer
    review**;
2.  **post-print**, which is the manuscript as **accepted after peer
    review** but not yet typeset as an article in the journal;
3.  and **typeset manuscript/publisher's manuscript**, which is the
    manuscript with the text after peer-review, **fully typeset**, as it
    appears in the journal.

<figure>
<a href="https://twitter.com/ryregier/status/1010915830496972808"><img
src="/img/blog-images/2019-06-04-rromeo/article_types.jpg"
alt="The different types of articles. Preprints, then Accepted Manuscript or Post-Print after Peer Review, then Publisher Version or Version of Record after Publication" /></a>
<figcaption><p><a
href="https://twitter.com/ryregier/status/1010915830496972808">figure</a>
by <a href="https://awayofhappening.wordpress.com/">Ryan Regier</a>,
with <a
href="https://commons.wikimedia.org/wiki/File:Book_designed_by_Benny_Forsberg_from_the_Noun_Project.svg">Book
icon</a> from <a href="https://thenounproject.com/foppa/">Benny
Forsberg</a>, <a
href="https://creativecommons.org/licenses/by/3.0/deed.en">CC-BY
3.0</a></p></figcaption>
</figure>

Some journals accept the archival only of the pre-print, while others
accept both pre-print and post-print, or even accept the archival of all
three versions!SHERPA/RoMEO's API lets you know what is the policy of a
journal using its name, or its ISSN, and whether restrictions apply such
as embargo periods before publicly archiving different manuscript
versions.

But even though the database is still updated, it seems the development
of the API stopped in 2013, which means it's [lacking some
functionalities](https://sherpa.ac.uk/romeo/API-Wishlist-2012-10.html)
and it does not always follow modern web standards.Because of this, we
could not always readily use R packages but we often needed to perform
small adjustments first.For example, it did not always use valid XML and
the character encoding was not declared in the HTTP headers, but in the
body of the document.Furthermore, the SHERPA/RoMEO is not
[RESTful](https://en.wikipedia.org/wiki/Representational_state_transfer)
and thus the queries were a little more complex to design.Fortunately,
the developers had written a [full
documentation](https://www.sherpa.ac.uk/romeo/apimanual.php) of all
different types of query we could run.

### rromeo: an R API client

rromeo lets you access basic information regarding the journal policies
in R.You can get the policy of a specific journal with its title using
the function `rr_journal_name()`:

::: {.highlight}
``` {.chroma tabindex="0"}
rromeo::rr_journal_name("Methods in Ecology and Evolution")
```
:::

``` {tabindex="0"}
                              title      issn romeocolour preprint 1 Methods in Ecology and Evolution 2041-210X      yellow      can    postprint    pdf pre_embargo post_embargo pdf_embargo 1 restricted cannot        <NA>    12 months        <NA>
```

In this example, we see that 'Methods in Ecology and Evolution' allows
the archival of pre-print manuscripts, post-print manuscripts (but with
restrictions) but it does not allow the archival of the typeset
manuscript (`pdf` column). A 12 months embargo is required before the
public archival of post-print manuscripts (`post_embargo` column).

You can also fetch the policy of several journals at once by matching
the beginning of the title via the argument `qtype = "starts"`:

::: {.highlight}
``` {.chroma tabindex="0"}
rromeo::rr_journal_name("Bird", qtype = "starts")
```
:::

``` {tabindex="0"}
4 journals match your query terms.Recursively fetching data from each journal. This may take some time...                            title      issn romeocolour preprint postprint1                   Bird Behavior 0156-1383        blue   cannot    cannot2 Bird Conservation International 0959-2709       green      can       can3                Bird Populations 1074-1755        <NA>     <NA>      <NA>4                      Bird Study 0006-3657       green      can       can     pdf pre_embargo post_embargo pdf_embargo1    can        <NA>         <NA>        <NA>2 cannot        <NA>         <NA>        <NA>3   <NA>        <NA>         <NA>        <NA>4 cannot        <NA>         <NA>        <NA>
```

Finally, rromeo can also retrieve publisher's information:

::: {.highlight}
``` {.chroma tabindex="0"}
rromeo::rr_publisher_name("Oxford University Press")
```
:::

``` {tabindex="0"}
   romeoid               publisher alias romeocolour preprint  postprint 1      55 Oxford University Press   OUP      yellow      can restricted       pdf 1 unclear
```

which gives you the general policy of the 'Oxford University Press'
regarding manuscript archival. Some restrictions may apply depending on
the precise branch of the publisher you plan to publish with, so you
should always double check before archiving your manuscripts.

You can get a full overview of rromeo capabilities by reading the
[introductory
vignette](https://docs.ropensci.org/rromeo/articles/rromeo.html).

### Developing an API package: first steps

We had no prior experience working with web technologies within R, and
this ended up begin a rich and fruitful experience for us.Our first step
was to choose the R package to perform the web requests.We picked
[httr](https://github.com/r-lib/httr) over rOpenSci's
[crul](https://github.com/ropensci/crul) package for its even
higher-level of use as httr hides even more details on how it handles
the query compared to crul.We made sure to follow the best practices
described in its ["Building API Packages"
vignette](https://cran.r-project.org/web/packages/httr/vignettes/api-packages.html).We
then naturally turned to the
[xml2](https://cran.r-project.org/package=xml2) package to parse the
resulting XML file, [as recommended in rOpenSci package development
book](https://devguide.ropensci.org/building.html#recommended-scaffolding).

We wanted to follow best development practices such as having unit tests
to check the behavior of functions in the package. But we didn't
intially know how to run unit tests that required an internet
connection. We discovered that the answer is **mocking**: storing
locally fake HTTP responses that *mimic* the API and use them to test
the functions in our package.Creating these fake responses is not an
easy task but fortunately, rOpensci's
[vcr](https://github.com/ropensci/vcr) package is exactly suited for
this task. vcr records requests and replays them during the tests (learn
more about it in [the technote about
vcr](/technotes/2018/05/25/vcr-http-request-cacahing/)). We used it in
all our tests as well as for caching the examples shown in the `README`
file. The [companion book on HTTP
testing](https://ropenscilabs.github.io/http-testing-book/) helped us
dive into the different options suitable to record our requests.

While working on rromeo, we realized that the level of details offered
to the user was a delicate balance: on one side, we ran the risk of not
being flexible enough to be useful and on the other side, the complexity
of the package would just become untractable. We were greatly inspired
by the many packages developed by [Scott
Chamberlain](/authors/scott-chamberlain/) that sometimes offer two
different interfaces for different kind of users (see
[rredlist](https://github.com/ropensci/rredlist) for example). This
strategy may be useful to offer different kind of granularity of
information on details depending on the future use of the API
information.

A similar issue lay with the license of the SHERPA/RoMEO data: the API
returns a [license notice](https://www.sherpa.ac.uk/romeoreuse.html)
with each request but we didn't want to flood the user with too much
information. We opted to include this information in the `CITATION` file
of the package accessible with the command `citation("rromeo")`. We
since then noticed other strategies in other packages, such as having a
[`DATA_USE`](https://github.com/ropensci/seaaroundus/blob/master/DATA_USE)
file in the root of the source repository and we are still thinking
about what the best choice is here.

### Developing an API package: gotchas and lessons learned

We were eager to learn but we made some mistakes in the process.
Thankfully, these were caught in [the review
process](https://github.com/ropensci/software-review/issues/285) before
we submitted the package for the first official release on CRAN.

For example, it is good practice to use an API key when working with
APIs, to let the owners track usage. rromeo provides several ways to set
up an API key [after registering
it](https://sherpa.ac.uk/romeo/apiregistry.php): a `key` argument in all
functions, setting up an environment variable in the session, using an
`.Renviron` or an `.Rprofile` file. These 4 options are summarized in
the [API key
vignette](https://docs.ropensci.org/rromeo/articles/setting_up_api_key.html).
Our first explanation on how to setup an API key was difficult to
understand and was located in the function help files. We thank both our
reviewers, [Philipp Ottolinger](https://github.com/ottlngr/) and [Bruna
Wundervald](https://github.com/brunaw), for encouraging us to write a
full vignette regarding API keys. Thanks to their comments we also wrote
the `rr_auth()` function that writes the key as an environment variable.
There may be room for improvement regarding the security of the API key
but the access to the API is probably not very sensitive in our case.

Similar to setting up an API key, setting a user-agent when doing
web-scraping is good practice to let the owners of the website/API from
which kind of software the requests come from. [Bruna
Wundervald](https://github.com/brunaw) pointed out that we had forgotten
to setup a user-agent even though it was specified in the [httr best
practices
vignette](https://cran.r-project.org/web/packages/httr/vignettes/api-packages.html).
We tweaked our requests using `httr::add_headers("user-agent" = ...)` to
add a user-agent with a custom defined string that links to the GitHub
page of the package and returns its version number.To check that the
user-agent was well defined we used the awesome website
<https://httpbin.org/> which is a simple HTTP request & response
service. It is very useful to test prototype queries and make sure you
get back what you wanted. We used <https://httpbin.org/user-agent> with
our custom specified header to make sure we had set up the user-agent
correctly. We do recommend using httpbin.org services to prototype
requests and test the passing of arguments.

Because all packages are in the end made for users, it is always
important to think about the interface you want to expose to your users.
We first had a single function `rr_journal_name()` with a Boolean
argument called `multiple`. Based on the value of this argument, the
result of `rr_journal_name()` could be wildly different. Both our
reviewers noticed it was unexpected from a naive user point of view and
suggested to split the function in two different functions that had
consistent output: `rr_journal_name()` would return all the information
regarding policies and `rr_journal_find()` would return simply the title
as well as the ISSN of the journal if found. The new outlook of
independent reviewers helped us take a step back regarding the functions
we were exposing to the users.

### Giving back to the community

As we mentioned previously, our main source of attraction towards
rOpenSci was its thriving community. We've always been convinced that
collaboration produced the best scientific output. Because of this, we
were glad to notice that our experience with rromeo could have ripple
effects and help other projects in the community.

Filing issues when you find a bug is an easy but efficient way to give
back to the community. [vcr](https://github.com/ropensci/vcr) and
[webmockr](https://github.com/ropensci/webmockr) are recent rOpenSci
packages that have mainly been used by a handful of developers until
now, one of them being rOpenSci co-founder [Scott
Chamberlain](/authors/scott-chamberlain/). Scott often uses crul in his
API packages while we used httr for rromeo. We were able to identify
issues with vcr and webmockr when used with httr but they were fixed
promptly ^[1](#fn:1){.footnote-ref
role="doc-noteref"}[2](#fn:2){.footnote-ref
role="doc-noteref"}[3](#fn:3){.footnote-ref role="doc-noteref"}^.

We also plan to contact the SHERPA/RoMEO API developers to let them know
we developed rromeo and have their feedback on the package.

### The future

rromeo is both on GitHub and CRAN now and seems quite stable. The
SHERPA/RoMEO API offers more data regarding paid open access policies
(<https://www.sherpa.ac.uk/romeo/PaidOA.php>) as well as other
restrictions on manuscript archival. We are still looking for a suitable
format to return this information to users. If you want to get involved,
we welcome contributions (look for the [issues tagged "help wanted" on
GitHub](https://github.com/ropensci/rromeo/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)).

We enjoyed the experience of contributing to rOpenSci, benefiting from
great recommendations by the editor and reviewers, as well as
discovering bugs in lesser used packages along the way.

We are now moving to [another package](https://github.com/Rekyt/rtaxref)
that we plan to submit to rOpenSci. It is also an API package so we'll
be using some of the knowledge we got from developing rromeo. We hope to
be back soon on the rOpenSci blog to talk about that ;)

### Acknowledgements

We would like to thank both of the reviewers [Philipp
Ottolinger](https://github.com/ottlngr/) and [Bruna
Wundervald](https://github.com/brunaw) as well as the editor [Scott
Chamberlain](/authors/scott-chamberlain/) for [their input to improve
the documentation and the
package](https://github.com/ropensci/software-review/issues/285).We also
want to mention Scott Chamberlain for building so many in tools to make
the interactions with API within R easier (vcr, webmockr, crul, as well
as many API packages that can be used as examples for developing your
own API package).

::: {.footnotes role="doc-endnotes"}

------------------------------------------------------------------------

1.  ::: {#fn:1}
    <https://github.com/ropensci/vcr/issues/91> [↩︎](#fnref:1){.footnote-backref
    role="doc-backlink"}
    :::

2.  ::: {#fn:2}
    <https://github.com/ropensci/vcr/issues/92> [↩︎](#fnref:2){.footnote-backref
    role="doc-backlink"}
    :::

3.  ::: {#fn:3}
    <https://github.com/ropensci/vcr/issues/93> [↩︎](#fnref:3){.footnote-backref
    role="doc-backlink"}
    :::
:::


We’ve been following rOpenSci’s work for a long time, and we use several packages on a daily basis for our scientific projects, especially taxize to clean species names, rredlist to extract species IUCN statuses or [treeio](many probs with this post) to work with phylogenetic trees.rOpensci is a perfect incarnation of a vibrant and diverse community where people learn and develop new ideas, especially regarding scientific packages.We’ve also

Rogue Scholar Beiträge

Access Publisher Copyright & Self-Archiving Policies via the 'SHERPA/RoMEO' API