Scott Chamberlain

In February this year I wroute about how many parameters functions should have, looking at some other languages, with a detailed look at R. On a related topic …  As I work on many R packages that are API clients for various web services, I began wondering: What is the best way to deal with API routes that have a lot of parameters?

API client design: how to deal with lots of parameters?

notes to self for next job hunt (some of which may be generally useful): don’t apply to general tech companies anymore for many reasons. heard back from very very few but that may be b/c I don’t know many people at general tech companies never been able to get through interviews; they’re presumably looking for computer science grads (not me) most of their missions are probably not stuff I’d be happy about at the end of the day.

Job searching notes

I recently had a use case at work where I wanted to check that file paths given in a Python script actually existed. These paths were in various GitHub repositories, so all I had to do was pull out the paths and check if they exist on GitHub. There were a few catches though.

Python, ast, and redbaron

My parents just found this email they had printed out from me from May 19, 2006, when I was 26. I chatted about some family stuff, then had this rambling string of weird thoughts below. I thought others might appreciate a good cringe - or cringy laugh - at my expense. It’s especially funny because I’m most def an atheist. I don’t know, those Tucson sunsets really are transformative.

Weird thoughts

I was working on a work task last week, and needed to filter out one instance of a class from a list of class instances. No matter how you do this speed doesn’t matter too much if you’re doing this operation once or a few times. However, I this operation needs to be done about 100K times each time the script runs - so speed definitely does matter in this case.

List comprehension vs. filter vs. key lookup

taxizedb arose from pain in using taxize when dealing with large amounts of data in a single request or doing a lot of requests of any data size. taxize works with remote data sources on the web, so there’s a number of issues that can slow the response down: internet speed, server response speed (was a response already cached or not; or do they even use caching), etc.

taxizedb: an update

I worked on a refactor of an R package at work the other day. Here’s some notes about that after doing the work. This IS NOT a best practices post - it’s just a collection of thoughts. For context, the package is an API client.

Refactoring notes

This blog is now using Hugo. Important - if you subscribe to the RSS for this blog you likely have to delete/remove the old one and add the new RSS link.

Moved to Hugo

I wrote the other day about overcoming an issue with Shiny. Another issue I ran into concurrently was about file inputs. The issue was that file inputs (i.e., shiny::fileInput) was difficult to clear.

Shiny file inputs

I recently opened an issue in a repository for a package I’m working on to think about potential footguns and how to avoid them. That word “footguns” got me thinking;

Avoiding the word footgun(s)

I’ve been working on an inherited Shiny app at work for the past few months. One of the many frustrating things about Shiny lately has been around buttons. Well, it wasn’t really about buttons, but that’s where it started. I noticed that a number of buttons - some having file inputs, some having text inputs - did not trigger every time, and I expected them to trigger every time no matter what.

Shiny button weirdness

TL;DR  In 6 months (end of November 2022) the CRAN Checks API https://cranchecks.info/ will be gone You can still get badges at https://badges.cranchecks.info You can use the new badges like: [![cran checks](https://badges.cranchecks.info/worst/dplyr.svg)](https://cran.r-project.org/web/checks/check_results_dplyr.html)  Find more details at https://github.com/sckott/cchecksbadges Sunsetting the CRAN Checks API   If you contribute an R package to

CRAN Checks API and Badges

It’s been interesting switching jobs with respect to programming languages. I used to write 95% R - now I write 95% Python.  I have been using Python for many years, but not seriously or getting paid either. I’ve learned alot in the first 6 months.  Some Python things learned: Functions and methods   I used to think functions and methods were the same thing. But during the last 6 months I learned that functions and methods are not the same.

Notes on Python

Soooo, my last job at Deck was amazing. I loved it. I was doing data engineer stuff there, mostly maintaining infrastructure for data pipelines. Everyone was great and the mission was amazing: helping Democrats win. Yet the company was shut down about a month ago, sending me on another job search, the 3rd since early/mid 2021.

Working at Fred Hutchinson Cancer Center

You’ve experienced an HTTP redirect (or URL redirect, or URL forwarding) even if you haven’t noticed. We all use browsers (I assume, since you are reading this), either on a phone or laptop/desktop computer. Browsers don’t show all the HTTP requests going on in the background, some of which are redirects.

Mocking HTTP redirects

\  Code is often arranged in packages for any given language. Packages are often cataloged in a package registry of some kind: NPM for node, crates.io for Rust, etc. For R, that registry is either CRAN or Bioconductor (for the most part).  CRAN has the concept of an archived package.

stories behind archived packages

Recology

I was working on a work task last week, and needed to filter out one instance of a class from a list of class instances. No matter how you do this speed doesn't matter too much if you're doing this operation once or a few times.

However, I this operation needs to be done about 100K times each time the script runs - so speed definitely does matter in this case.

First, I naively started off with using `filter()`{.language-plaintext .highlighter-rouge}. When that lead to waiting more than ten minutes, I read that list comprehensions are faster.

Second, I tried list comprehensions. Also waited more then ten minutes and gave up again.

Last, I thought perhaps it would work to make a dictionary where the keys are the things I need to filter on and the values the class instances. This was the answer! Super fast.

Now some examples of what I'm talking about.

Make a class called `Stuff`{.language-plaintext .highlighter-rouge}.

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
class Stuff:
  def __init__(self, letter):
    self.letter = letter
    super(Stuff, self).__init__()
  def __repr__(self):
        return "<{} ({})>".format(self.__class__.__name__, self.letter)

x = Stuff('s')
x
#> <Stuff (s)>
x.letter
#> 's'
```
:::
:::

Make a list of instances of the class `Stuff`{.language-plaintext .highlighter-rouge}

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
import string

lst = []
for x in string.ascii_lowercase:
    lst.append(Stuff(x))
lst
#> [<Stuff (a)>, <Stuff (b)>, <Stuff (c)>, <Stuff (d)>, <Stuff (e)>, <Stuff (f)>, <Stuff (g)>, <Stuff (h)>, <Stuff (i)>, <Stuff (j)>, <Stuff (k)>, <Stuff (l)>, <Stuff (m)>, <Stuff (n)>, <Stuff (o)>, <Stuff (p)>, <Stuff (q)>, <Stuff (r)>, <Stuff (s)>, <Stuff (t)>, <Stuff (u)>, <Stuff (v)>, <Stuff (w)>, <Stuff (x)>, <Stuff (y)>, <Stuff (z)>]
len(lst)
#> 26
```
:::
:::

List comprehension: This is how I did the list comprehension method. Filter the list `lst`{.language-plaintext .highlighter-rouge} where some attibute matched some value.

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
[x for x in lst if x.letter == 'f']
#> [<Stuff (f)>]
```
:::
:::

Filter: This is how I did the filter method. Filter the list `lst`{.language-plaintext .highlighter-rouge} where some attibute matched some value.

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
list(filter(lambda x: x.letter == 'f', lst))
#> [<Stuff (f)>]
```
:::
:::

And here's the dictionary approach. Here, I first make a dictionary via `dict(zip())`{.language-plaintext .highlighter-rouge} where the keys are some attribute of each instance. You can lookup by key.

A major difference/drawback of this approach is that it only works if there's only one match per key because dictionaries don't allow duplicate keys.

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
lst_map = dict(zip([w.letter for w in lst], lst))
lst_map['f']
#> <Stuff (f)>
```
:::
:::

And better yet, use `.get()`{.language-plaintext .highlighter-rouge} so you don't run into a `KeyError`{.language-plaintext .highlighter-rouge} when the key doesn't exist

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
lst_map.get('f')
#> <Stuff (f)>
lst_map.get('5')
#> (returns None)
```
:::
:::

What about the timings:

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
from timeit import timeit
n = 100000
time_list_comp = timeit("[x for x in lst if x.letter == 'f']", number=n, globals=globals())
time_filter = timeit("list(filter(lambda x: x.letter == 'f', lst))", number=n, globals=globals())
time_dict = timeit("lst_map['f']", number=n, globals=globals())
time_dict_get = timeit("lst_map.get('f')", number=n, globals=globals())

round(time_list_comp, 3)
#> 0.088
round(time_filter, 3)
#> 0.134
round(time_dict, 3)
#> 0.002
round(time_dict_get, 3)
#> 0.003
```
:::
:::

For bracketed lookup, the list comprehension is **39** times slower, and the filter is **59** times slower.

For the `get()`{.language-plaintext .highlighter-rouge} lookup, the list comprehension is **26** times slower, and the filter is **39** times slower.


It's been interesting switching jobs with respect to programming languages. I used to write 95% R - now I write 95% Python.

I have been using Python for many years, but not seriously or getting paid either. I've learned alot in the first 6 months.

Some Python things learned:

## Functions and methods

I used to think functions and methods were the same thing. But during the last 6 months I learned that functions and methods are not the same. Well, they're not that different. A function outside a class is just called a [function](https://docs.python.org/3/glossary.html#term-function) while a function inside a class is called a [method](https://docs.python.org/3/glossary.html#term-method). They could be exactly the same and do the same thing, but one is outside a class and the other inside a class.

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
class Stuff(object):
  def things():
    return 5

def things():
    return 5

Stuff.things()
# 5
things()
# 5
```
:::
:::

## .sort and sorted

`.sort`{.language-plaintext .highlighter-rouge} called on an object changes the object in place and `sorted()`{.language-plaintext .highlighter-rouge} creates a new object.

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
x = [4,1,7,2,6,5,3]
z = x.sort()
# nothing returned, z = None
sorted(x)
# [1, 2, 3, 4, 5, 6, 7]
```
:::
:::

## context managers

I'd surely used a [context manager](https://docs.python.org/3/glossary.html#term-context-manager) in Python before but didn't realize what was happening. The code base I work in uses many [with](https://docs.python.org/3/reference/compound_stmts.html#with) statements and these are used with context managers like:

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
with EXPRESSION as TARGET:
    SUITE
```
:::
:::

In the above case I often create a connection to a database using the `with`{.language-plaintext .highlighter-rouge} statement, then once the block is exited, the connection is cleaned up.

## imports

Coming from R, it's so nice in Python to be able to import specific functions, classes, etc. rather than having to load an entire file or package in R. In addition, `as`{.language-plaintext .highlighter-rouge} in Python imports is really nice to have.

Python

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
from x import y
import pandas as pd
```
:::
:::

R

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
require(x)
library(x)
```
:::
:::

Sometimes I run into circular import issues in Python (as I did in R), which I've yet to find a neat solution to sorting out.


You've experienced an [HTTP redirect](https://en.wikipedia.org/wiki/URL_redirection) (or URL redirect, or URL forwarding) even if you haven't noticed. We all use browsers (I assume, since you are reading this), either on a phone or laptop/desktop computer. Browsers don't show all the HTTP requests going on in the background, some of which are redirects. Redirection is used for various reasons, including to prevent broken links when web pages are moved, for privacy protection, to allow multiple domains to refer to a single web page, and more.

The easiest way to know if you hit a redirect is to look at the [HTTP status code](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes). Status codes in the 3xx series are mostly about URL redirection. The most common you will see are 301 (moved permanently), 302 (moved temporarily), and 303 (see other URI; usually in a "Location" response header).

When making HTTP requests in R, redirects are generally handled automatically by the three HTTP clients ([curl](https://jeroen.cran.dev/curl/), [crul](https://github.com/ropensci/crul), [httr](https://github.com/r-lib/httr)). That is, if a 300 series code is detected, all three clients will go to the next URI and so on until there are no more redirects. Automatically following redirects may not be default behavior elsewhere (e.g., `crul`{.language-plaintext .highlighter-rouge} command line tool doesn't follow redirects by default), so beware.

HTTP redirects become more tricky when we have to mock them in unit tests or other similar situations. I'll cover the various tools for doing this in R.

## Redirects

First, I'll show how redirects work with three major HTTP clients:

[curl](https://jeroen.cran.dev/curl/)

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
library(curl)
h <- curl::new_handle()
handle_setopt(h, followlocation=0L)
out <- curl_fetch_memory("https://hb.opencpu.org/redirect/3", handle = h)
curl::parse_headers(out$headers, multiple = TRUE)
```
:::
:::

[crul](https://github.com/ropensci/crul)

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
library(crul)
con <- HttpClient$new("https://hb.opencpu.org/redirect/3")
res <- con$get()
length(res$response_headers_all)
#> [1] 4
```
:::
:::

[httr](https://github.com/r-lib/httr)

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
library(httr)
z <- GET("https://hb.opencpu.org/redirect/3")
length(z$all_headers)
#> [1] 4
```
:::
:::

## Mocking redirects

If you want to mock HTTP redirects, you can do so with the [webmockr](https://github.com/ropensci/webmockr) package. Why would you want to mock redirects?

Here's one use case: Say you have a library/package interacting with a web resource that you interact with. You want to add some unit tests for a route that responds with one or more redirects. You'd prefer not to perform real HTTP requests against the remote service for one reason or another (e.g., extreme rate limiting); and some would say it's best not to test with real HTTP requests b/c you want to test the functionality of the package, NOT the remote server with which it interacts.

In the following, we re-create what happens in real HTTP requests - but just status codes and the `location`{.language-plaintext .highlighter-rouge} response header.

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
library(webmockr)
library(crul)
webmockr::enable()
```
:::
:::

Make a single stub with each redirect response with `to_return()`{.language-plaintext .highlighter-rouge}

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
stub_request("get", "https://hb.opencpu.org/redirect/3") %>%
  to_return(status = 302, headers = list(location = "/relative-redirect/2")) %>%
  to_return(status = 302, headers = list(location = "/relative-redirect/1")) %>%
  to_return(status = 302, headers = list(location = "/get")) %>%
  to_return(status = 200, headers = list(location = "hooray, all done!"))
```
:::
:::

Then make four different requests to `https://hb.opencpu.org/redirect/3`{.language-plaintext .highlighter-rouge}:

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
con <- crul::HttpClient$new(url = "https://hb.opencpu.org")
con$get("redirect/3")$response_headers
#> $location
#> [1] "/relative-redirect/2"
con$get("redirect/3")$response_headers
#> $location
#> [1] "/relative-redirect/1"
con$get("redirect/3")$response_headers
#> $location
#> [1] "/get"
con$get("redirect/3")$response_headers
#> $location
#> [1] "hooray, all done!"
```
:::
:::

This isn't ideal because it doesn't reflect how the real HTTP request equivalent happens.

Alernatively, you could set it up like this, with four separate stubs:

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
stub_request("get", "https://hb.opencpu.org/redirect/3") %>%
  to_return(status = 302, headers = list(location = "/relative-redirect/2"))

stub_request("get", "https://hb.opencpu.org/relative-redirect/2") %>%
  to_return(status = 302, headers = list(location = "/relative-redirect/1"))

stub_request("get", "https://hb.opencpu.org/relative-redirect/1") %>%
  to_return(status = 302, headers = list(location = "/get"))

stub_request("get", "https://hb.opencpu.org/get") %>%
  to_return(status = 200, headers = list(location = "hooray, all done!"))
```
:::
:::

Then make each request in turn to each successive URL:

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
con <- crul::HttpClient$new(url = "https://hb.opencpu.org")
con$get("redirect/3")$response_headers
#> $location
#> [1] "/relative-redirect/2"
con$get("relative-redirect/2")$response_headers
#> $location
#> [1] "/relative-redirect/1"
con$get("relative-redirect/1")$response_headers
#> $location
#> [1] "/get"
con$get("get")$response_headers
#> $location
#> [1] "hooray, all done!"
```
:::
:::

## Faking real redirects

[vcr](https://github.com/ropensci/vcr) is built on top of webmockr, but instead of returning stubbed responses and not allowing real HTTP requests, vcr stores real HTTP request/response and uses them on all subsequent matching HTTP requests.

I wrote this back in March 2021 - and was waiting to figure out how to deal with redirects in [vcr](https://github.com/ropensci/vcr) before finishing this post - see [vcr issue #220](https://github.com/ropensci/vcr/issues/220). I still have and may never get to that issue. If you are interested, please do stop by vcr and make a pull request to get it fixed. The major issue is that vcr stores only the first HTTP response in a redirect chain, rather than the last HTTP response - as I would expect.


In February this year I wroute about [how many parameters functions should have](https://recology.info/2020/02/how-many-parameters/), looking at some other languages, with a detailed look at R. On a related topic ...

As I work on many R packages that are API clients for various web services, I began wondering: What is the best way to deal with API routes that have a lot of parameters?

The general programming wisdom I've seen is that a function should have no more than 3-4 parameters (e.g., [this long SO thread](https://stackoverflow.com/questions/174968/how-many-parameters-are-too-many), or [this one](https://softwareengineering.stackexchange.com/questions/331803/techniques-for-minimising-number-of-function-arguments)). So should one do anything different from a normal function when that function is connecting to a web API route with a lot of parameters? I've not found very much spilled ink on this exact topic, but I'll discuss what I have found.

## Use cases?

A Software Engineering StackExchange thread [How to handle many arguments in an API wrapper?](https://softwareengineering.stackexchange.com/questions/196895/how-to-handle-many-arguments-in-an-api-wrapper) had a couple ideas. One idea is to consider use cases, and then make separate functions covering those use cases. This might work, but I haven't explored it thoroughly for a real API route yet. Pondering on it though I doubt this would work since you'd have to pre-emptively think about all the different scenarios users might dream up, which seems like a fools errand.

## Builder pattern

Another concept brought up in the thread mentioned above was the [Builder pattern](https://en.wikipedia.org/wiki/Builder_pattern). It's hard for me to understand the idea in abstract - [here's a nicer discussion of this in Ruby](https://medium.com/kkempin/builder-design-pattern-in-ruby-dfa2d557ff1b).

This is a good concept to know about, but I don't think is appropriate for the issue at hand, of how to handle many API parameters.

## Named parameter map

Another idea in that thread was to use a named parameter map. In R this would look something like this (imagine a lot more parameters though):

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
foo <- function(args) {
  get("/some-api-route", args)
}
api_args <- list(query = "*:*", limit = 10)
my_args <- modifyList(api_args, list(query = "bears", limit = 300))
foo(my_args)
```
:::
:::

That is, the above would replace this:

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
foo <- function(query = "*:*", limit = 10) {
  get("/some-api-route", list(query = query, limit = limit))
}
foo(query = "*:*", limit = 10)
```
:::
:::

So in the first code block the function no longer has a lot of parameters in it. The drawback of this in R (and I'm sure is similar in other languages) is that users lose the autocomplete helpers that most modern IDE's and text editors have - helping users type less and quickly get a tip on what each parameter is intended to do and importantly (if the developer has documented the function well) what types the parameters expects and what (if any) options there are to pass to the parameter.

Of course a user can "just" read the docs to figure out what each parameter expects, but it sure can save a lot of time if the help is right there in the tooltips of the IDE/text editor. In addition, in R there's automated checking that parameters in functions are also documented, which is nice for making sure parameters and docs don't get out of sync. You'd lose this by using a parameter map - though you could document the parameter map - and perhaps wire together some custom code to make sure the parameters in the parameter map are all handled by the function. This does seem like a lot of fuss though compared to simply having the parameters in the function itself.

This approach probably becomes more attractive if a client has many functions that take the same parameters - in which a named parameter map could handle the parameters and any logic behind checking those parameters.

## Include no parameters in the function

In other words: Pass all parameters on to the API w/o including any of them in the function - i.e., let the API handle any problems in parameters.

Another approach I've not seen written about but that I've seen in code is having a rather lite wrapper around an API and letting the API itself sort out any problems due to user inputs.

An example is the [gh R package](https://gh.r-lib.org/), a client for the GitHub API. For query parameters you can pass in named parameters through the ellipsis `...`{.language-plaintext .highlighter-rouge}, all of which are passed as query parameters. The gh package does no checking of these parameters (that I know of); simply passes them to the GitHub API. The GitHub API happens to apparently ignore invalid (silently drop) parameters and invalid valuses of parameters (here, "stuff" is an invalid value for the `page`{.language-plaintext .highlighter-rouge} parameter).

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
x <- gh("GET /users/{username}/repos", username = "gaborcsardi", page = "stuff")
length(x)
#> 30
```
:::
:::

I don't hate this solution, but I don't love it either. This approach is highly dependent on a well designed API that fails gracefully, with informative error messages and with correct status codes, etc. I would say most APIs are not as nice as GitHub's, at least in the scientific API space in which I work.

One plus side of this approach is the R package gh only has one parameter (`...`{.language-plaintext .highlighter-rouge}) to handle all query parameters, so you do solve the too many query parameters problem.

Another upside to this approach is you do not have to keep up with any changes in parameters on each API route - for example, an API route could drop one parameter, and add another, and the R client wouldn't have to change anything (assuming the change in parameters wasn't associated with a change that breaks code in the client).

A major downside of this approach is that the user often has to mount a time-consuming expedition to figure out what parameters are accepted. Some API clients may document them, and some will simply direct users to the web APIs docs. I think this part alone makes this solution (include no parameters in the function) not a good one since the user experience can be so bad if the documentation is not good. And all developers know its much easier for their docs to get out of date than their code.

## Include some parameters in the function

Another approach is to define some query parameters in the function, and handle all others via R's ellipsis (`...`{.language-plaintext .highlighter-rouge}) - or similar in other languages. I've seen this relatively often and have used it myself. It's often used when there's a clear smaller set of important parameters - those can be put in the function as named parameters. And then there's a long tail of other parameters that the maintainer thinks are not likely to be used very often. Those can be looked up by the user in the API docs for whatever API the client interacts with.

An example of this is the rOpenSci package [rtweet](https://docs.ropensci.org/rtweet/) - a client for the Twitter API. In the [search_tweets() function](https://docs.ropensci.org/rtweet/reference/search_tweets.html#arguments) there are a half dozen or so named parameters in the function, but then the ellipsis handles all other parameters.

The drawback to this approach is that no two APIs behave the same way. In the case of Twitter they silently ignore/drop parameters they do not support (same as the GitHub API, see above). For example:

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
library(rtweet)
search_tweets("hillaryclinton", n = 3, foo = "bar")
```
:::
:::

Works just fine even though `foo`{.language-plaintext .highlighter-rouge} is absolutely not a parameter supported by the Twitter API. They must ignore parameters they don't support. This is same behavior as the GitHub API we saw above.

In the case of Twitter and GitHub one might want to raise errors on unsupported parameters client side in rtweet to avoid any use confusion of parameters being silently dropped.

## Grouping similar parameters together

Many threads (e.g., [this one](https://softwareengineering.stackexchange.com/a/352676/329940)) suggest that similar parameters could be grouped together to reduce the number of parameters passed to a function. For example, if a function has the parameters `latitude`{.language-plaintext .highlighter-rouge} and `longitude`{.language-plaintext .highlighter-rouge} you could group those into a single parameter called e.g., `coordinates`{.language-plaintext .highlighter-rouge}.

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
# Original function, each parameter separate
foo <- function(latitude, longitude) {
    # do something with latitude/longitude
    latitude
    longitude
}

# Modified function, grouping the two parameters into one
foo <- function(coordinates) {
    # do something with latitude/longitude
    coordinates$latitude
    coordinates$longitude
}
```
:::
:::

Though I've not tried this approach myself, it might be a good compromise between a function not handling any parameters (i.e., just passing all to the web API unhandled) and handling every parameter individually.

## Closing thoughts

The benefit of documenting API query parameters in a client package is that you can tell the user what each parameters expects in language they can understand. That is, if you simply direct users to the docs for the web API with which the client interacts, the API docs could be not very good and/or specify types expected that the user may not understand. In addition, there may be edge cases or similar with some parameters that are not documented in the API docs but that you can document in the client docs for each parameter.

I would say the vast majority of web API clients I use that do succeed in having very few parameters also have docs in which it's a nightmare trying to figure out what parameters each method accepts. That is, the pursuit of very few parameters at least is correlated with a very poor user experience - in my experience.


> Update on 2021-02-09: I've archived 8 more packages. Post below updated

\

Code is often arranged in packages for any given language. Packages are often cataloged in a package registry of some kind: NPM for node, crates.io for Rust, etc. For R, that registry is either [CRAN](https://cran.r-project.org/) or [Bioconductor](https://bioconductor.org/) (for the most part).

CRAN has the concept of an archived package. That is, the namespace for a package (`foo`{.language-plaintext .highlighter-rouge}) is still in the registry (and can not be used again), but the package is archived - no longer gets updated and checks I think are no longer performed.

We rarely hear the stories behind how software gets laid to rest. What are the most common reasons for software to be abandoned?

## My CRAN archived packages

First, my archived CRAN packages:

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
library(pkgsearch)
library(dplyr)
library(data.table)
library(tibble)
x = cran_events(releases = FALSE, archivals = TRUE, limit = 4000L)
res = lapply(x, function(w)
  tibble(pkg=w$name, maintainer=w$package$Maintainer))
df = rbindlist(res, use.names = TRUE, fill = TRUE)
df = as_tibble(df)
scott <- filter(df, grepl("chamberlain", maintainer, ignore.case = TRUE)) %>%
  select(pkg) %>% 
  data.frame()
scott
#>            pkg
#> 1      originr
#> 2       geoaxe
#> 3         lawn
#> 4     pleiades
#> 5       geoops
#> 6          rif
#> 7     rbraries
#> 8        ccafs
#> 9     rjsonapi
#> 10       rdpla
#> 11 seaaroundus
#> 12    crevents
#> 13      etseed
#> 14      rtimes
#> 15   rsunlight
#> 16        nneo
#> 17     binomen
#> 18        solr
#> 19      enigma
#> 20         alm
#> 21    ropensnp
#> 22      govdat
#> 23  spoccutils
#> 24     rgauges
```
:::
:::

I have 24 archived packages on CRAN.

## Stories

The following are brief stories of why each package was archived on CRAN.

-   **spoccutils**: was a package of utility functions that didn't quite fit within the scope of another package [spocc](https://github.com/ropensci/spocc). It was renamed to [mapr](https://github.com/ropensci/mapr). I suppose I could have asked CRAN to change the name, hmmm
-   **[rgauges](https://github.com/ropensci-archive/rgauges)**: was a client for the [Gaug.es](https://get.gaug.es/) website analytics API - we started the package to gather data on visitors to the rOpenSci website. Eventually we stopped using Gaug.es and then it didn't make sense to maintain the package, so it was archived.
-   **[alm](https://github.com/ropensci-archive/alm)**: was a client for a generic article-level metrics web service framework called Lagotto. In the early days of rOpenSci we were quite engaged with the community of folks working on article-level metrics. If I remember correctly, Lagotto usage slowed down and wasn't used much so I gave up on maintaining `alm`{.language-plaintext .highlighter-rouge}
-   **ropensnp**: was a client for the service [OpenSNP](https://opensnp.org/). There were other sources of SNP data that I thought would be nice to access all from one package; so a new package ([rsnps](https://github.com/ropensci/rsnps)) was created and incorporated functions for OpenSNP and `ropensnp`{.language-plaintext .highlighter-rouge} was archived.
-   **solr**: was a package client for the [Apache Solr](https://lucene.apache.org/solr/) database. At some point it got a major overhual and I decided to change the package name to [solrium](https://github.com/ropensci/solrium).
-   **[binomen](https://github.com/ropensci/binomen)**: aim of the package was functions for creating taxonomic classes/objects and functions for manipulating taxonomic data, sort of like dplyr. Evolution of ideas in `binomen`{.language-plaintext .highlighter-rouge} gave way to a new package [taxa](https://github.com/ropensci/taxa), now maintained by [Zach Foster](https://github.com/zachary-foster).
-   **[nneo](https://github.com/ropensci-archive/nneo)**: was a client for APIs for [NEON](https://data.neonscience.org/data-api/) data. At some point I stumbled upon someone else from NEON making essentially the same package, so I archived mine.
-   **[etseed](https://github.com/ropensci-archive/etseed)**: was a package for interacting with the distributed key-value store [etcd](https://etcd.io/). I was on a kick at the time of making R packages for databases, and saw a missing package I thought. After getting familiar with etcd, I realized I would probably never use it myself, and further, it probably didn't make sense to interact with etcd from R anyway.
-   **[crevents](https://github.com/ropensci-archive/crevents)**: [Crossref](https://www.crossref.org/) mints DOIs for scholarly articles (among other works). A neat service they started was collecting and making searchable the "events" on DOIs - that is, the links pointing to DOIs, e.g., from Twitter, etc. The service at some point became very unreliable (was often down), so the package was archived.
-   **[seaaroundus](https://github.com/ropensci-archive/seaaroundus)**: [Seaaroundus](http://www.seaaroundus.org/) maintains fisheries and fisheries-related data. I was helping maintain an R package for the API, but it was a difficult one to maintain, and most users simply sent emails requesting dumps of data anyway - so the package was archived.
-   **[rdpla](https://github.com/ropensci-archive/rdpla)**: The [Digital Public Library of America](https://dp.la/) is a very cool organization that is similar to Crossref in a way, in that they centralize metadata about "things"; metadata on museum collections for DPLA and scholarly works for Crossref. I figured many researchers would enjoy being able to easily get metadata from DPLA for research on museum collections. In the end not many people used the package.
-   **[rjsonapi](https://github.com/ropensci-archive/rjsonapi)**: [JSON:API](https://jsonapi.org/) is a cool idea - a sort of specification for building APIs in JSON. REST APIs are incredibly variable - this is an attempto standardize it a bit. I thought perhaps JSON:API would be adopted widely and that an R client would be useful for consuming JSON:API services - however, I've seen only very few APIs using JSON:API.
-   **[originr](https://github.com/ropensci-archive/originr)**: The idea with `originr`{.language-plaintext .highlighter-rouge} originally was to centralize in one package tools for biologists to get data on "nativity" of species in their studies: are species X and Y native to A, B, and C countries. I was lucky to get a collaborator ([Ignasi Bartomeus](https://github.com/ibartomeus)) to help on that package. The package simply was not used much at all, and the data sources used were very flaky, making for a buggy user experience.
-   **[geoaxe](https://github.com/ropensci-archive/geoaxe)**: This package always had a very narrow scope. It was created to solve a problem in two widely used packages: [rgbif](https://github.com/ropensci/rgbif) and [spocc](https://github.com/ropensci/spocc). In both of those packages we needed to take user input of a Well-known text (WKT) string representing a polygon in which the user wanted to search for something in a remote data source. And with that WKT we would chop up the polygon into smaller polygons to be submitted in multiple (spatial) requests instead of one very large (spatial) request. I didn't want to depend on the heavy `rgdal`{.language-plaintext .highlighter-rouge} depenency, so I cooked up `geoaxe`{.language-plaintext .highlighter-rouge} that only used `sp`{.language-plaintext .highlighter-rouge} and `rgeos`{.language-plaintext .highlighter-rouge}. `geoaxe`{.language-plaintext .highlighter-rouge} lasted for many years, but there's better tools out there now, so it was archived.
-   **[lawn](https://github.com/ropensci-archive/lawn)**: This package started out when I was trying to see what we could do in spatial R packages, specifically around GeoJSON. Jeroen had recently created the [V8](https://github.com/jeroen/v8) package so you could leverage bundled Javascript libraries in R. [turf.js](https://turfjs.org/) was a neat project for Javascript for spatial analysis and built in a modular, approachable manner - I thought. So collaborator [Jeff Hollister](https://github.com/jhollist) and I wrapped turf.js in R. It was not used that much - and especially not used much after the rise of sf and related "tidy" spatial packages, leading to archival.
-   **[geoops](https://github.com/sckott/geoops)**: This package was a follow on from `lawn`{.language-plaintext .highlighter-rouge}, in exploring more GeoJSON focused work. I wanted to learn how to make an R package mostly out of C++, with just a thin layer of R on top. Like `lawn`{.language-plaintext .highlighter-rouge}, `geoops`{.language-plaintext .highlighter-rouge} was not used much at all as far as I could tell, maybe partly because it focused only on GeoJSON - but this was also during the rise of `sf`{.language-plaintext .highlighter-rouge} and friends. I moved it to my own personal account just as a little C++ in R playground in case I want to brush up on that.
-   **[pleiades](https://github.com/ropensci-archive/pleiades)**: I don't remember how or why this package started. It was a client for Pleiades (<https://pleiades.stoa.org/home>), a database of historical geographic information about the ancient world. It was used very, very little as far as I could tell - so was archived.
-   **[rif](https://github.com/ropensci-archive/rif)**: This package was started after a some exploration of what R work we (rOpenSci) could do in the neuroscience field. It was a client for the Neuroscience Information Framework (<https://neuinfo.org/>), a database of neuroscience information. Another package not used - so was archived.
-   **[rbraries](https://github.com/ropensci-archive/rbraries)**: This package started because I thought rOpenSci would make use of the metadata around R package downloads/etc. that <https://libraries.io/> collected. We never ended up using the data - and no one seemed to use the package - so was archived.
-   **[ccafs](https://github.com/ropensci-archive/ccafs)**: A client for Climate Change, Agriculture, and Food Security (CCAFS) General Circulation Models (GCM) data (<http://www.ccafs-climate.org/>). All I could find for motivation for this package was an email with someone where they shared a few links to different sources of climate data - then apparently this package was born shortly thereafter. It wasn't used much at all - so was archived.

The following four packages were all R clients for sources of government open data - see the organization [rOpenGov](https://github.com/ropengov) for R packages on government data.

-   **govdat**: was split into two packages (`rsunlight`{.language-plaintext .highlighter-rouge} and `rtimes`{.language-plaintext .highlighter-rouge}) and `govdat`{.language-plaintext .highlighter-rouge} was archived
-   **[rsunlight](https://github.com/rOpenGov/rsunlight)**: was a client for many APIs of the organization [Sunlight Labs](https://sunlightfoundation.com/api/) - part of the reason for archiving this package was the disintegration of Sunlight Labs, which made the previously sensible organization of many APIs into one R package not sensible anymore. Also, government data was considered out of scope for our work at rOpenSci.
-   **[rtimes](https://github.com/rOpenGov/rtimes)**: was a client for a number of the government data APIs from the New York Times. One reason for abandoning this package was that NYT almost never responded to questions/feedback on their APIs. Another reason was the aforementioned focus of rOpenSci.
-   **[enigma](https://github.com/rOpenGov/enigma)**: was a client for the Enigma API - the company I think was first focused on making open government data easier to access - as well as data on companies. I didn't really use the package at all though, and there wasn't much usage of the package, so archived it.

As a summary of the lists above, a list of the major reasons each package was archived:

-   Not used
    -   rjsonapi
    -   rdpla
    -   etseed
    -   enigma
    -   rgauges
    -   rtimes
    -   originr
    -   geaoxe
    -   pleiades
    -   rif
    -   rbraries
    -   ccafs
-   The fall of the GeoJSON-verse ^[1](#fn:1){.footnote rel="footnote"}^
    -   lawn
    -   geoops
-   Bad/retired service
    -   rtimes
    -   seaaroundus
    -   crevents
    -   alm
-   Name change
    -   solr
    -   ropensnp
    -   spoccutils
-   Evolution to new package
    -   binomen
    -   govdat
-   Duplicated work
    -   nneo
-   Out of scope
    -   rsunlight

Footnotes:

::: {.footnotes role="doc-endnotes"}
1.  ::: {#fn:1}
    although, `geojsonsf`{.language-plaintext .highlighter-rouge} is very successful so I think my projects were just crap I guess :) [↩](#fnref:1){.reversefootnote role="doc-backlink"}
    :::
:::


[taxizedb](https://github.com/ropensci/taxizedb) arose from pain in
using [taxize](https://github.com/ropensci/taxize) when dealing with
large amounts of data in a single request or doing a lot of requests of
any data size. [taxize](https://github.com/ropensci/taxize) works with
remote data sources on the web, so there's a number of issues that can
slow the response down: internet speed, server response speed (was a
response already cached or not; or do they even use caching), etc.

The idea with [taxizedb](https://github.com/ropensci/taxizedb) was to
allow users to do the same things as taxize allows, but much faster by
accessing the entire database for a data source on their own computer.
The previous versions of taxizedb used a variety of different databases
(MySQL/MariaDB, PostgreSQL, SQLite), so the technical barrier to entry
was pretty high. In the newest version just released, we've drastically
simplified the database situation, among other things.

[taxadb](https://github.com/ropensci/taxadb) was developed as an
alternative approach to taxizedb and should also be considered when
dealing with taxonomic names. It takes a different approach for the
data, with tabular files hosted on GitHub releases, but is similar in
that after downloading the data is put into a SQL database, SQLite by
default (with other options for databases). taxadb user facing functions
are different from those in taxizedb, and largely don't overlap.

taxizedb quick links:

- [taxizedb repo](https://github.com/ropensci/taxizedb)
- [taxizedb on cran](https://cloud.r-project.org/web/packages/taxizedb/)
- [taxizedb docs](https://ropensci.github.io/taxizedb/)

Install the latest version, if you don't get `v0.2.0` with

<div>

<div>

    install.packages("taxizedb")

</div>

</div>

then use

<div>

<div>

    install.packages("taxizedb", repos = "https://dev.ropensci.org")

</div>

</div>

Load the package

<div>

<div>

    library(taxizedb)

</div>

</div>

  

## All SQLite!

SQLite is shipped in nearly every device these days, so taxizedb now
uses only SQLite for the database backend for each data source. Every
person that installs taxizedb should have SQLite already installed. In
addition, there's no usernames/passwords/ports to setup with. How we've
accomplished this is partly through automation:

- NCBI: SQLite built within R from tabular files
- ITIS: they provide a SQLite dump
- Plantlist: is no longer updated; we build a SQLite manually from csv
  files
- COL: a SQLite is built once a day via GitHub Actions
- GBIF: a SQLite is built once a day via GitHub Actions
- Wikidata: SQLite built within R from a tabular file
- World Flora Online: SQLite built within R from a tabular file

Some of the databases have indices to speed up queries, making them a
bit larger relative to no indices, but these days most people likely are
willing to use up a little more disk space on their computer to have
faster queries.

  

## New data sources

Three new data sources were added:

- [NCBI taxonomy](https://www.ncbi.nlm.nih.gov/taxonomy) - all of this
  work was done by [Zebulun Arendsee](https://github.com/arendsee)
- [World Flora Online](http://www.worldfloraonline.org/) - the
  replacement for The Plant List
- [Wikidata](https://zenodo.org/record/1213477) - the table
  `wikidata-taxon-info`, extracted taxon objects from Wikidata, last
  updated April 2018, on Zenodo

  

## New functions: taxize equivalents

Three new high level functions matching those in taxize were added:
`children`, `classification`, `downstream`. The taxize version of those
functions are still good for smaller requests, but with larger requests,
its probably best to use taxizedb. The most common problem where taxize
becomes frustrating is with `downstream` where a user wants all species
within a high taxonomic rank like phylum. The original work for these
functions was done by [Zebulun Arendsee](https://github.com/arendsee).

Here's a comparison of taxize vs. taxizedb with `downstream` - getting
all species within the genus *Bombus* (bumble bees)

<div>

<div>

    id_tx <- taxize::get_tsn("Bombus")
    system.time(taxize::downstream(id_tx, db = "itis", downto = "species"))
    #>    user  system elapsed 
    #>   2.144   0.130  20.533

    id_txdb <- taxizedb::name2taxid('Bombus', db = "itis")
    system.time(taxizedb::downstream(id_txdb, db = "itis", downto = "species"))
    #>    user  system elapsed 
    #>   0.132   0.051   0.186

</div>

</div>

  

In addition, three new "mapping" functions were added that are similar
to those in taxize, but with different names: `name2taxid` (scientific
or common name to taxonomy ID); `taxid2name` (taxonomy ID to scientific
name); `taxid2rank` (taxonomy ID to rank).

We saw `name2taxid` above. Below we get the taxonomic ID for COL, ITIS
and GBIF for *Bombus*

<div>

<div>

    name2taxid('Bombus', db = "col")
    #> [1] "3993765"
    name2taxid('Bombus', db = "itis")
    #> [1] "154397"
    name2taxid('Bombus', db = "gbif")
    #> [1] "1340278"

</div>

</div>

Get the scientific name from a taxonomic ID

<div>

<div>

    taxid2name(3993765, db = "col")
    #> [1] "Bombus"
    taxid2rank(3993765, db = "col")
    #> [1] "genus"

</div>

</div>

These functions are quite fast too:

<div>

<div>

    x <- taxize::names_list(rank = "species", size = 10000L)
    system.time(name2taxid(x, db = "gbif", out_type = "summary"))
    #>    user  system elapsed 
    #>   0.096   0.206   1.053

</div>

</div>

  

## Thoughts?

Get in touch if you have any feedback at
<https://github.com/ropensci/taxizedb/issues>
