Scott Chamberlain

In February this year I wroute about how many parameters functions should have, looking at some other languages, with a detailed look at R. On a related topic …  As I work on many R packages that are API clients for various web services, I began wondering: What is the best way to deal with API routes that have a lot of parameters?

API client design: how to deal with lots of parameters?

notes to self for next job hunt (some of which may be generally useful): don’t apply to general tech companies anymore for many reasons. heard back from very very few but that may be b/c I don’t know many people at general tech companies never been able to get through interviews; they’re presumably looking for computer science grads (not me) most of their missions are probably not stuff I’d be happy about at the end of the day.

Job searching notes

I recently had a use case at work where I wanted to check that file paths given in a Python script actually existed. These paths were in various GitHub repositories, so all I had to do was pull out the paths and check if they exist on GitHub. There were a few catches though.

Python, ast, and redbaron

My parents just found this email they had printed out from me from May 19, 2006, when I was 26. I chatted about some family stuff, then had this rambling string of weird thoughts below. I thought others might appreciate a good cringe - or cringy laugh - at my expense. It’s especially funny because I’m most def an atheist. I don’t know, those Tucson sunsets really are transformative.

Weird thoughts

I was working on a work task last week, and needed to filter out one instance of a class from a list of class instances. No matter how you do this speed doesn’t matter too much if you’re doing this operation once or a few times. However, I this operation needs to be done about 100K times each time the script runs - so speed definitely does matter in this case.

List comprehension vs. filter vs. key lookup

taxizedb arose from pain in using taxize when dealing with large amounts of data in a single request or doing a lot of requests of any data size. taxize works with remote data sources on the web, so there’s a number of issues that can slow the response down: internet speed, server response speed (was a response already cached or not; or do they even use caching), etc.

taxizedb: an update

I worked on a refactor of an R package at work the other day. Here’s some notes about that after doing the work. This IS NOT a best practices post - it’s just a collection of thoughts. For context, the package is an API client.

Refactoring notes

This blog is now using Hugo. Important - if you subscribe to the RSS for this blog you likely have to delete/remove the old one and add the new RSS link.

Moved to Hugo

I wrote the other day about overcoming an issue with Shiny. Another issue I ran into concurrently was about file inputs. The issue was that file inputs (i.e., shiny::fileInput) was difficult to clear.

Shiny file inputs

I recently opened an issue in a repository for a package I’m working on to think about potential footguns and how to avoid them. That word “footguns” got me thinking;

Avoiding the word footgun(s)

I’ve been working on an inherited Shiny app at work for the past few months. One of the many frustrating things about Shiny lately has been around buttons. Well, it wasn’t really about buttons, but that’s where it started. I noticed that a number of buttons - some having file inputs, some having text inputs - did not trigger every time, and I expected them to trigger every time no matter what.

Shiny button weirdness

TL;DR  In 6 months (end of November 2022) the CRAN Checks API https://cranchecks.info/ will be gone You can still get badges at https://badges.cranchecks.info You can use the new badges like: [![cran checks](https://badges.cranchecks.info/worst/dplyr.svg)](https://cran.r-project.org/web/checks/check_results_dplyr.html)  Find more details at https://github.com/sckott/cchecksbadges Sunsetting the CRAN Checks API   If you contribute an R package to

CRAN Checks API and Badges

It’s been interesting switching jobs with respect to programming languages. I used to write 95% R - now I write 95% Python.  I have been using Python for many years, but not seriously or getting paid either. I’ve learned alot in the first 6 months.  Some Python things learned: Functions and methods   I used to think functions and methods were the same thing. But during the last 6 months I learned that functions and methods are not the same.

Notes on Python

Soooo, my last job at Deck was amazing. I loved it. I was doing data engineer stuff there, mostly maintaining infrastructure for data pipelines. Everyone was great and the mission was amazing: helping Democrats win. Yet the company was shut down about a month ago, sending me on another job search, the 3rd since early/mid 2021.

Working at Fred Hutchinson Cancer Center

You’ve experienced an HTTP redirect (or URL redirect, or URL forwarding) even if you haven’t noticed. We all use browsers (I assume, since you are reading this), either on a phone or laptop/desktop computer. Browsers don’t show all the HTTP requests going on in the background, some of which are redirects.

Mocking HTTP redirects

\  Code is often arranged in packages for any given language. Packages are often cataloged in a package registry of some kind: NPM for node, crates.io for Rust, etc. For R, that registry is either CRAN or Bioconductor (for the most part).  CRAN has the concept of an archived package.

stories behind archived packages

Recology

You've experienced an [HTTP redirect](https://en.wikipedia.org/wiki/URL_redirection) (or URL redirect, or URL forwarding) even if you haven't noticed. We all use browsers (I assume, since you are reading this), either on a phone or laptop/desktop computer. Browsers don't show all the HTTP requests going on in the background, some of which are redirects. Redirection is used for various reasons, including to prevent broken links when web pages are moved, for privacy protection, to allow multiple domains to refer to a single web page, and more.

The easiest way to know if you hit a redirect is to look at the [HTTP status code](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes). Status codes in the 3xx series are mostly about URL redirection. The most common you will see are 301 (moved permanently), 302 (moved temporarily), and 303 (see other URI; usually in a "Location" response header).

When making HTTP requests in R, redirects are generally handled automatically by the three HTTP clients ([curl](https://jeroen.cran.dev/curl/), [crul](https://github.com/ropensci/crul), [httr](https://github.com/r-lib/httr)). That is, if a 300 series code is detected, all three clients will go to the next URI and so on until there are no more redirects. Automatically following redirects may not be default behavior elsewhere (e.g., `crul`{.language-plaintext .highlighter-rouge} command line tool doesn't follow redirects by default), so beware.

HTTP redirects become more tricky when we have to mock them in unit tests or other similar situations. I'll cover the various tools for doing this in R.

## Redirects

First, I'll show how redirects work with three major HTTP clients:

[curl](https://jeroen.cran.dev/curl/)

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
library(curl)
h <- curl::new_handle()
handle_setopt(h, followlocation=0L)
out <- curl_fetch_memory("https://hb.opencpu.org/redirect/3", handle = h)
curl::parse_headers(out$headers, multiple = TRUE)
```
:::
:::

[crul](https://github.com/ropensci/crul)

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
library(crul)
con <- HttpClient$new("https://hb.opencpu.org/redirect/3")
res <- con$get()
length(res$response_headers_all)
#> [1] 4
```
:::
:::

[httr](https://github.com/r-lib/httr)

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
library(httr)
z <- GET("https://hb.opencpu.org/redirect/3")
length(z$all_headers)
#> [1] 4
```
:::
:::

## Mocking redirects

If you want to mock HTTP redirects, you can do so with the [webmockr](https://github.com/ropensci/webmockr) package. Why would you want to mock redirects?

Here's one use case: Say you have a library/package interacting with a web resource that you interact with. You want to add some unit tests for a route that responds with one or more redirects. You'd prefer not to perform real HTTP requests against the remote service for one reason or another (e.g., extreme rate limiting); and some would say it's best not to test with real HTTP requests b/c you want to test the functionality of the package, NOT the remote server with which it interacts.

In the following, we re-create what happens in real HTTP requests - but just status codes and the `location`{.language-plaintext .highlighter-rouge} response header.

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
library(webmockr)
library(crul)
webmockr::enable()
```
:::
:::

Make a single stub with each redirect response with `to_return()`{.language-plaintext .highlighter-rouge}

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
stub_request("get", "https://hb.opencpu.org/redirect/3") %>%
  to_return(status = 302, headers = list(location = "/relative-redirect/2")) %>%
  to_return(status = 302, headers = list(location = "/relative-redirect/1")) %>%
  to_return(status = 302, headers = list(location = "/get")) %>%
  to_return(status = 200, headers = list(location = "hooray, all done!"))
```
:::
:::

Then make four different requests to `https://hb.opencpu.org/redirect/3`{.language-plaintext .highlighter-rouge}:

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
con <- crul::HttpClient$new(url = "https://hb.opencpu.org")
con$get("redirect/3")$response_headers
#> $location
#> [1] "/relative-redirect/2"
con$get("redirect/3")$response_headers
#> $location
#> [1] "/relative-redirect/1"
con$get("redirect/3")$response_headers
#> $location
#> [1] "/get"
con$get("redirect/3")$response_headers
#> $location
#> [1] "hooray, all done!"
```
:::
:::

This isn't ideal because it doesn't reflect how the real HTTP request equivalent happens.

Alernatively, you could set it up like this, with four separate stubs:

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
stub_request("get", "https://hb.opencpu.org/redirect/3") %>%
  to_return(status = 302, headers = list(location = "/relative-redirect/2"))

stub_request("get", "https://hb.opencpu.org/relative-redirect/2") %>%
  to_return(status = 302, headers = list(location = "/relative-redirect/1"))

stub_request("get", "https://hb.opencpu.org/relative-redirect/1") %>%
  to_return(status = 302, headers = list(location = "/get"))

stub_request("get", "https://hb.opencpu.org/get") %>%
  to_return(status = 200, headers = list(location = "hooray, all done!"))
```
:::
:::

Then make each request in turn to each successive URL:

::: {.language-r .highlighter-rouge}
::: highlight
``` highlight
con <- crul::HttpClient$new(url = "https://hb.opencpu.org")
con$get("redirect/3")$response_headers
#> $location
#> [1] "/relative-redirect/2"
con$get("relative-redirect/2")$response_headers
#> $location
#> [1] "/relative-redirect/1"
con$get("relative-redirect/1")$response_headers
#> $location
#> [1] "/get"
con$get("get")$response_headers
#> $location
#> [1] "hooray, all done!"
```
:::
:::

## Faking real redirects

[vcr](https://github.com/ropensci/vcr) is built on top of webmockr, but instead of returning stubbed responses and not allowing real HTTP requests, vcr stores real HTTP request/response and uses them on all subsequent matching HTTP requests.

I wrote this back in March 2021 - and was waiting to figure out how to deal with redirects in [vcr](https://github.com/ropensci/vcr) before finishing this post - see [vcr issue #220](https://github.com/ropensci/vcr/issues/220). I still have and may never get to that issue. If you are interested, please do stop by vcr and make a pull request to get it fixed. The major issue is that vcr stores only the first HTTP response in a redirect chain, rather than the last HTTP response - as I would expect.
