Maëlle Salmon

Scott Chamberlain

rOpenSci

Stefanie Butland

Last year we reported on [the joy of using commonmark and xml2 to
parseMarkdown content, like the source of this website built
withHugo](/technotes/2018/09/05/commonmark/), in particular [to
extractlinks](https://ropensci.org/technotes/2018/09/05/commonmark/#urls-parsing),at
the time merely to count them. How about we go a bit further and usethe
same approach to find links to be fixed? In this tech note we
shallreport our experience using R to find broken/suboptimal links and
fixthem.

## What is a bad URL?

We tackled a few URL issues:

- We had used [absolute links (using our domain name) instead
  ofrelativelinks](https://stackoverflow.com/questions/2005079/absolute-vs-relative-urls).`https://ropensci.org/blog/`
  should be `/blog/`.

- Some internal and external links were broken, but we did not knowwhich
  ones.

- A few links were short links (`bit.ly/blabla`) whereas it's best
  tostore the actual link because the short link could break too.

- Some links were http links, although the same https link might workand
  would be preferred over http for security.

- There were links to ropensci.github.io documentation websites thatcan
  be replaced with links to [our brand-new
  docsserver](/technotes/2019/06/07/ropensci-docs/).

## Please read if you try this at "home/"

There are three main ingredients to our website spring/fall cleaning:
Rtools, elbow grease and *version control*! Most changes happened in
abranch, and although one can't possibly look in detail at a diff of
morethan one hundred files, we tried to be as careful as possible.

## From absolute to relative links

To remove the absolute links, we resorted to using regular expressions.

::: {.highlight}
``` {.chroma tabindex="0"}
library("magrittr")# Identify the Markdown files to be examinedmds <- fs::dir_ls("content", recurse = TRUE, glob = "*.md")mds <- mds[!grepl("\\/tutorials\\/", mds)]# Function to fix each file if neededfix_ropensci <- function(filepath){  readLines(filepath) -> text    # We only edit files that had the issue  if (any(grepl("http(s)?\\:\\/\\/ropensci\\.org\\/", text))){    text %>%      stringr::str_replace_all("http(s)?\\:\\/\\/ropensci\\.org\\/", "/") %>%      writeLines(filepath)  }}purrr::walk(mds, fix_ropensci)
```
:::

Voilà!

## Broken URLs

Now, what about the links that do not link to anything? We started
byextracting all links together with the relevant file paths.

::: {.highlight}
``` {.chroma tabindex="0"}
library("magrittr")website_source <- "/home/maelle/Documents/ropensci/roweb2"mds <- fs::dir_ls(website_source, recurse = TRUE, glob = "*.md")mds <- mds[!grepl("\\/tutorials\\/", mds)]get_links <- function(filepath){  readLines(filepath) %>%    glue::glue_collapse(sep = "\n") %>%    commonmark::markdown_html(normalize = TRUE,                              extensions = TRUE) %>%    xml2::read_html() %>%    xml2::xml_find_all("//a") %>%    xml2::xml_attr("href") -> urls    tibble::tibble(filepath = filepath,                 url = urls)}all_urls <- purrr::map_df(mds, get_links)all_urls <- all_urls  %>%  dplyr::mutate(url = stringr::str_remove_all(url, "#.*"),                url = stringr::str_remove(url, "\\/$"))all_urls
```
:::

    ## # A tibble: 14,234 x 2##    filepath                                            url                      ##    <chr>                                               <chr>                    ##  1 /home/maelle/Documents/ropensci/roweb2/content/aut… https://adamhsparks.netl…##  2 /home/maelle/Documents/ropensci/roweb2/content/aut… https://aldocompagnoni.w…##  3 /home/maelle/Documents/ropensci/roweb2/content/aut… http://robitalec.ca      ##  4 /home/maelle/Documents/ropensci/roweb2/content/aut… https://alison.rbind.io  ##  5 /home/maelle/Documents/ropensci/roweb2/content/aut… https://dobb.ae          ##  6 /home/maelle/Documents/ropensci/roweb2/content/aut… https://thestudyofthehou…##  7 /home/maelle/Documents/ropensci/roweb2/content/aut… https://annakrystalli.me ##  8 /home/maelle/Documents/ropensci/roweb2/content/aut… https://paleantology.com…##  9 /home/maelle/Documents/ropensci/roweb2/content/aut… https://aurielfournier.g…## 10 /home/maelle/Documents/ropensci/roweb2/content/aut… https://faculty.washingt…## # … with 14,224 more rows

We chose a different method to find those within and outside of
ourwebsite.

### Broken internal URLs

When building a Hugo website, one gets a sitemap, which is basically
acollection of links to all the pages of the website. [If an
internallink is not in the sitemap, it does not
exist.](https://masalmon.eu/404)

We generated the sitemap from within the website folder to extractlinks.

::: {.highlight}
``` {.chroma tabindex="0"}
cwd <- getwd()setwd(website_source)p <- processx::process$new("hugo", args = "server", echo = TRUE)
```
:::

    ## Running hugo server

::: {.highlight}
``` {.chroma tabindex="0"}
Sys.sleep(120)localhost <- "http://localhost:1313"browseURL(localhost)paste0(localhost, "/sitemap.xml") %>%  xml2::read_xml() %>%  xml2::xml_ns_strip() %>%  xml2::xml_find_all("//loc") %>%  xml2::xml_text() %>%  stringr::str_remove_all(localhost) %>%  stringr::str_remove("\\/$") -> linksp$kill()
```
:::

    ## [1] TRUE

::: {.highlight}
``` {.chroma tabindex="0"}
setwd(cwd)head(links)
```
:::

    ## [1] "/authors/scott-chamberlain"         "/tags/api"                         ## [3] "/authors"                           "/tags/http"                        ## [5] "/technotes/2019/12/11/http-testing" "/tags/mocking"

So these are the existing internal links. We could also have
extractedthem using [the multi-request features
ofcurl](https://github.com/jeroen/curl/blob/master/examples/sitemap.R).

Let's now extract the internal links we used in the content.

::: {.highlight}
``` {.chroma tabindex="0"}
all_urls %>%  dplyr::filter(!grepl("^http", url)) ->   internal_urlshead(internal_urls)
```
:::

    ## # A tibble: 6 x 2##   filepath                                             url                      ##   <chr>                                                <chr>                    ## 1 /home/maelle/Documents/ropensci/roweb2/content/blog… /community               ## 2 /home/maelle/Documents/ropensci/roweb2/content/blog… /community               ## 3 /home/maelle/Documents/ropensci/roweb2/content/blog… /blog/2013/05/10/introdu…## 4 /home/maelle/Documents/ropensci/roweb2/content/blog… /about                   ## 5 /home/maelle/Documents/ropensci/roweb2/content/blog… /community               ## 6 /home/maelle/Documents/ropensci/roweb2/content/blog… /contact

So, what are the missing ones? We used the code below to identify
themand then we [manually fixed or
removedthem](https://github.com/ropensci/roweb2/pull/534/commits/7715dc022141888b4f064cc44fe50b29b031c755).

::: {.highlight}
``` {.chroma tabindex="0"}
 internal_urls %>%   dplyr::filter(!url %in% links)
```
:::

### Broken external URLs

To identify broken external URLs, we
ran[`crul::ok()`](https://docs.ropensci.org/crul/reference/ok.html) on
allof them and created a big spreadsheet of URLs to look at.

::: {.highlight}
``` {.chroma tabindex="0"}
external_urls <- dplyr::anti_join(all_urls, internal_urls,                                   by = c("filepath", "url"))unique_urls <- unique(external_urls[, "url"])ok <- memoise::memoise(  ratelimitr::limit_rate(crul::ok,                          ratelimitr::rate(1, 1)))get_ok <- function(url){  message(url)  ok(url)}unique_urls <- unique_urls %>%  dplyr::group_by(url) %>%  dplyr::summarise(ok = get_ok(url))external_urls <- dplyr::left_join(external_urls, unique_urls,                                  by = "url")external_urls <- dplyr::arrange(external_urls, url)parse_one_post <- function(path){  if (grepl("\\_index", path)){    return(NULL)  }  lines <- suppressWarnings(readLines(path, encoding = "UTF-8"))  yaml <- blogdown:::split_yaml_body(lines)$yaml  yaml <- glue::glue_collapse(yaml, sep = "\n")  yaml <- yaml::yaml.load(yaml)    meta <- tibble::tibble(date = anytime::anydate(yaml$date),                         author = yaml$authors,                         title = yaml$title,                          software_peer_review = "Software Peer Review" %in% yaml$tags,                         type = dplyr::if_else(grepl("\\/blog\\/", path),                                               "blog post", "tech note"),                         filepath = path)    meta}info <- purrr::map_df(mds[grepl("blog", mds)|grepl("technotes",mds)], parse_one_post)info <- dplyr::group_by(info, filepath) %>%  dplyr::summarize(date = date[1],                   author = toString(author),                   title = title[1],                   type = type[1])bad_urls <- dplyr::filter(external_urls, !ok)bad_urls <- dplyr::left_join(bad_urls, info, by = "filepath")readr::write_csv(bad_urls, "urls.csv")
```
:::

From that spreadsheet hundreds of links were examined **manually**!
Whenthere was a replacement link, we used it thanks to a code looping
overall links. For the about 50 links without replacement, we amended
theposts by hand to make sure to take context into account
(e.g. removingthe link vs. removing the whole sentence presenting it).

There were quite a few false positives i.e. actually valid URLs.
Thislead to some edits
in[`crul::ok()`](https://docs.ropensci.org/crul/reference/ok.html) and
thefollowing wisdom:

Sometimes you'll get an error for the HEAD request but not the
GETrequest.

::: {.highlight}
``` {.chroma tabindex="0"}
# use get verb instead of headcrul::ok("http://animalnexus.ca")
```
:::

    ## [1] FALSE

::: {.highlight}
``` {.chroma tabindex="0"}
crul::ok("http://animalnexus.ca", verb = "get")
```
:::

    ## [1] TRUE

Sometimes you'll need an user-agent whose name does not contain
"curl",which the default user-agent of crul contains (`crul:::make_ua()`
islibcurl/7.58.0 r-curl/4.3 crul/0.9.1.9991).

::: {.highlight}
``` {.chroma tabindex="0"}
# some urls will require a different useragent string# they probably regex the useragent stringcrul::ok("https://doi.org/10.1093/chemse/bjq042")
```
:::

    ## GnuTLS recv error (-54): Error in the pull function.## [1] FALSE

::: {.highlight}
``` {.chroma tabindex="0"}
crul::ok("https://doi.org/10.1093/chemse/bjq042", verb = "get", useragent = "foobar")
```
:::

    ## [1] TRUE

## From short to long links

We only identified short links using the bit.ly service. We found
thecorresponding link by running the function below. There were
actuallyonly 4 short links so that was quick.

::: {.highlight}
``` {.chroma tabindex="0"}
get_long <- function(url){  crul::HttpClient$new(url)$get()$url}get_long("http://bit.ly/2JfrzmE")
```
:::

    ## [1] "https://www.timeanddate.com/worldclock/fixedtime.html?msg=rOpenSci+Community+Call+on+Reproducible+Research+with+R&iso=20190730T09&p1=791&ah=1"

## http vs https

> HTTPS: HTTP + security
> [pic.twitter.com/pkk7ZVzjz3](https://t.co/pkk7ZVzjz3)
>
> --- 🔎Julia Evans🔍 (@b0rk) [August 9,
> 2019](https://twitter.com/b0rk/status/1159812119099060224?ref_src=twsrc%5Etfw)

We proceeded as previously when checking external links, except we
usedbetter settings for `crul::ok()`.

::: {.highlight}
``` {.chroma tabindex="0"}
http <- dplyr::filter(all_urls, grepl("http\\:", url))http <- dplyr::mutate(http, https = sub("http\\:", "https:", url))unique_urls <- unique(http[, "https"])ok <- memoise::memoise(  ratelimitr::limit_rate(crul::ok,                          ratelimitr::rate(1, 1)))get_ok <- function(url){  message(url)  ok(url, verb = "get", useragent = "Maëlle Salmon checking links")}unique_urls <- unique_urls %>%  dplyr::group_by(https) %>%  dplyr::summarise(ok = get_ok(https))http <- dplyr::left_join(http, unique_urls, by = "https")httpsok <- dplyr::filter(http, ok)modify_url <- function(index, df = httpsok) {  row <- df[index,]  readLines(row$filepath) %>%    stringr::str_replace_all(row$url, row$https) %>%    writeLines(row$filepath)  }purrr::walk(seq_len(nrow(httpsok)), modify_url)
```
:::

You can [browse the
relatedPR](https://github.com/ropensci/roweb2/pull/582/). Note that in
theabove, we could have
used[`urltools`](https://github.com/Ironholds/urltools/) to parse URLs
andextract their scheme (http or https).

### docs.ropensci.org

To replace some ropensci.github.io links with docs.ropensci.org links,we
used the brute force approach below (there were only about 80
suchlinks).

::: {.highlight}
``` {.chroma tabindex="0"}
dotgithub <- dplyr::filter(all_urls, urltools::domain(url) == "ropensci.github.io")make_docs_url <- function(url, ropensci_pkgs = ropensci_pkgs) {  message(url)  newurl <- url  urltools::domain(newurl) <- "docs.ropensci.org"    if (crul::ok(newurl, verb = "get", useragent = "Maëlle Salmon checking links")) {    return(newurl)  } else {    return(url)  }}dotgithub <- dotgithub %>%  dplyr::group_by(url) %>%  dplyr::mutate(newurl = make_docs_url(url))modify_url <- function(index, df = dotgithub) {  row <- df[index,]  readLines(row$filepath) %>%    stringr::str_replace_all(row$url, row$newurl) %>%    writeLines(row$filepath)  }purrr::walk(seq_len(nrow(dotgithub)), modify_url)
```
:::

## Conclusion

In this tech note we saw how to use a combination of regularexpressions,
commonmark, xml2 and crul to identify links to be fixed inMarkdown
content. For html content, check out [the experimental checkerpackage by
François Michonneau](https://github.com/fmichonneau/checker).For
packages, have a look at [Bob Rudis'
RStudioadd-in](https://github.com/hrbrmstr/hrbraddins#whats-in-the-tin).

Some of the issues we fixed, like using relative rather than
absolutelinks, and not storing shortlinks, could be avoided in the
future bystricter URL guidelines for new content. We also plan to stop
usingClick here links, refer to [this page about why Click here links
arebad](https://webaccess.berkeley.edu/ask-pecan/click-here).

Now, a remaining issue is the frequency at which URL cleaning
shouldoccur. In [our dev guide](https://devguide.ropensci.org/), we
cleanlinks [before
eachrelease](https://devguide.ropensci.org/bookreleaseissue.html#bookreleaseissue),but
this website has no such schedule, so let's hope we remember toclean
URLs once in a while. Maybe some old pages could also be ["archived"
like this example](https://kb.iu.edu/d/abwp). When do *you* clean URLs
in your content,and how?


Last year we reported on the joy of using commonmark and xml2 to parseMarkdown content, like the source of this website built withHugo, in particular to extractlinks,at the time merely to count them. How about we go a bit further and usethe same approach to find links to be fixed? In this tech note we shallreport our experience using R to find broken/suboptimal links and fixthem.What is a bad URL?

We cleaned our website URLs with R!

## webmockr

`webmockr` is an R library for stubbing and setting expectations on HTTP
requests.It is a port of the Ruby gem
[webmock](https://github.com/bblimke/webmock).

`webmockr` works by plugging in to another R package that does HTTP
requests. It currently only works with
[crul](https://github.com/ropensci/crul) right now, but we plan to add
support for [curl](https://github.com/jeroen/curl) and
[httr](https://github.com/r-lib/httr) later.

`webmockr` has the following high level features:

- Stubbing HTTP requests at low http client level
- Setting expectations on HTTP requests
- Matching requests based any combination of HTTP method (e.g.,
  GET/POST/PUT), URI (i.e., URL), request headers and body
- Will soon integrate with [vcr](https://github.com/ropensci/vcr) so
  that you can cache real HTTP responses - and easily integrate with
  `testthat`

`webmockr` has been on CRAN for a little while now, but I've recently
made some improvements and am nearing anotherCRAN release, which is also
preparation for a first release of the related
[vcr](https://github.com/ropensci/vcr) package. The following is arun
down of major features and how to use the package, including a bit on
testing at the end.

## Similar art in other languages

`webmockr` was most closely inspired by Ruby's
[webmock](https://github.com/bblimke/webmock), but there are others out
there, e.g,. [HTTPretty](https://github.com/gabrielfalcao/HTTPretty) and
[pook](https://github.com/h2non/pook) for Python, from which we'll steal
ideas as time allows.

## Installation

`webmockr` is on CRAN, so you can install that version

::: {.highlight}
``` {.chroma tabindex="0"}
install.packages("webmockr")
```
:::

But I've been making some improvements, so you'll probably want the dev
version:

Install some dependencies

::: {.highlight}
``` {.chroma tabindex="0"}
if (!requireNamespace("remotes", quietly=TRUE)) install.packages("remotes")remotes::install_github(c("ropensci/vcr@webmockit", "ropenscilabs/fauxpas"))
```
:::

Install `webmockr`

::: {.highlight}
``` {.chroma tabindex="0"}
remotes::install_github("ropensci/webmockr")
```
:::

Load `webmockr`

::: {.highlight}
``` {.chroma tabindex="0"}
library(webmockr)
```
:::

## Using webmockr

After loading `webmockr`

::: {.highlight}
``` {.chroma tabindex="0"}
library(webmockr)
```
:::

`webmockr` is loaded but not "turned on". At this point `webmockr`
doesn'tchange anything about your HTTP requests.

Once you turn on `webmockr`:

::: {.highlight}
``` {.chroma tabindex="0"}
webmockr::enable()
```
:::

`webmockr` will now by default **NOT** allow real HTTP requests from the
httplibraries that adapters are loaded for (right now only `crul`).

Next, you'll want to think about stubbing requests

## Stubbing requests

Stubbing requests simply refers to the act of saying "I want all
HTTPrequests that match this pattern to return this thing". The
"thing"can be made up of a lot of different things.

First, load `crul` and enable `webmockr`

::: {.highlight}
``` {.chroma tabindex="0"}
library(crul)webmockr::enable()
```
:::

You can stub requests based on HTTP method and uri

::: {.highlight}
``` {.chroma tabindex="0"}
stub_request("get", "https://httpbin.org/get")#> <webmockr stub> #>   method: get#>   uri: https://httpbin.org/get#>   with: #>     query: #>     body: #>     request_headers: #>   to_return: #>     status: #>     body: #>     response_headers: #>   should_timeout: FALSE#>   should_raise: FALSE
```
:::

The returned object is of class `StubbedRequest`, with attributes
andaccessors - which you usually won't use yourself - although we use
theobject internally within `webmockr`. The object has a print method
thatsummarizes your stub, including the HTTP method, the URI, if you
setany patterns for query parameters, the query body, or request
headers,and whether you set any expectations on the response,
includingstatus code, body or response headers. Last, there's booleans
for whetheryou set a timeout or raise expectation (more on that below).

You can check the stub registry at any time to see what stubsyou have:

::: {.highlight}
``` {.chroma tabindex="0"}
stub_registry()#> <webmockr stub registry> #>  Registered Stubs#>    get: https://httpbin.org/get
```
:::

> protip: you can clear all stubs at any time by running
> `stub_registry_clear()`

Once we have the stub, we can then do a request with `crul`

::: {.highlight}
``` {.chroma tabindex="0"}
x <- HttpClient$new(url = "https://httpbin.org")x$get('get')#> <crul response> #>   url: https://httpbin.org/get#>   request_headers: #>     User-Agent: libcurl/7.54.0 r-curl/3.1 crul/0.5.1.9210#>     Accept-Encoding: gzip, deflate#>     Accept: application/json, text/xml, application/xml, */*#>   response_headers: #>   status: 200
```
:::

A response is returned that is of the same structure as a normal`crul`
response, but no actual HTTP request was performed. Theresponse object
is a bit different from a real HTTP response in thatwe don't have
response headers, but you can set an expectation ofresponse headers, and
even use real response headers you'veretrieved in real HTTP requests if
you like.

Once we've made a request we can take a peek into the requestregistry:

::: {.highlight}
``` {.chroma tabindex="0"}
request_registry()#> <webmockr request registry> #>   Registered Requests#>   GET https://httpbin.org/get   #>      with headers {#>        User-Agent: libcurl/7.54.0 r-curl/3.1 crul/0.5.0, #>        Accept-Encoding: gzip, deflate, #>        Accept: application/json, text/xml, application/xml, */*#>      } was made 1 times
```
:::

(the printing behavior above is manual - if you run this you'll see
oneline for each request - will make some prettier printing behavior
later)

We can see that a request to `https://httpbin.org/get` was made
withcertain request headers and was made 1 time. If we make that
requestagain the registry will then say 2 times. And so on.

You can also set the request headers that you want to match on.If you do
this the request has to match the HTTP method, the URI,and the certain
request headers you set.

::: {.highlight}
``` {.chroma tabindex="0"}
stub_request("get", "https://httpbin.org/get") %>%  wi_th(headers = list('User-Agent' = 'libcurl/7.51.0 r-curl/2.6 crul/0.3.6',                        'Accept-Encoding' = "gzip, deflate"))#> <webmockr stub> #>   method: get#>   uri: https://httpbin.org/get#>   with: #>     query: #>     body: #>     request_headers: User-Agent=libcurl/7.51.0 r-curl/2.6 crul/0.3.6, Accept-Encoding=gzip, deflate#>   to_return: #>     status: #>     body: #>     response_headers: #>   should_timeout: FALSE#>   should_raise: FALSE
```
:::

You can set the query parameters in a stub as well:

::: {.highlight}
``` {.chroma tabindex="0"}
stub_request("get", "https://httpbin.org/get") %>%  wi_th(    query = list(hello = "world"))#> <webmockr stub> #>   method: get#>   uri: https://httpbin.org/get#>   with: #>     query: hello=world#>     body: #>     request_headers: #>   to_return: #>     status: #>     body: #>     response_headers: #>   should_timeout: FALSE#>   should_raise: FALSE
```
:::

### Stubbing responses

Up until now the stubs we've created have only set what to match on,
buthave not set what to return. With `to_return` you can set response
headers,response body, and response HTTP status code. You don't match
requestson these three things, but rather they determine what's returned
in theresponse.

Here, we'll state that we want an HTTP status of 418 returned:

::: {.highlight}
``` {.chroma tabindex="0"}
stub_request("get", "https://httpbin.org/get") %>%    to_return(status = 418)#> <webmockr stub> #>   method: get#>   uri: https://httpbin.org/get#>   with: #>     query: #>     body: #>     request_headers: #>   to_return: #>     status: 418#>     body: #>     response_headers: #>   should_timeout: FALSE#>   should_raise: FALSE
```
:::

### Stubbing HTTP exceptions

Sometimes its useful in your stubs to say that you expect a particular
HTTPerror/exception.

`webmockr` has two functions for this:

- `to_raise`: raise any HTTP exception. pass in any http exception from
  the[fauxpas](https://github.com/ropenscilabs/fauxpas) package
- `to_timeout`: raise an HTTP timeout exception. this is it's own
  functionsince timeout exceptions are rather common and one may want to
  use them often

Here, with `to_raise` we pass in an HTTP exception, in this case the
`HTTPBadRequest`exception:

::: {.highlight}
``` {.chroma tabindex="0"}
library(fauxpas)stub_request("get", "https://httpbin.org/get?a=b") %>%     to_raise(fauxpas::HTTPBadRequest)#> <webmockr stub> #>   method: get#>   uri: https://httpbin.org/get?a=b#>   with: #>     query: #>     body: #>     request_headers: #>   to_return: #>     status: #>     body: #>     response_headers: #>   should_timeout: FALSE#>   should_raise: HTTPBadRequestx <- HttpClient$new(url = "https://httpbin.org")x$get('get', query = list(a = "b"))#> Error: Bad Request (HTTP 400).#>  - The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications.
```
:::

With `to_timeout` a matched request to our stub will then return a
timeout error:

::: {.highlight}
``` {.chroma tabindex="0"}
stub_request("post", "https://httpbin.org/post") %>%     to_timeout()#> <webmockr stub> #>   method: post#>   uri: https://httpbin.org/post#>   with: #>     query: #>     body: #>     request_headers: #>   to_return: #>     status: #>     body: #>     response_headers: #>   should_timeout: TRUE#>   should_raise: FALSEx <- HttpClient$new(url = "https://httpbin.org")x$post('post')#> Error: Request Timeout (HTTP 408).#>  - The client did not produce a request within the time that the server was prepared to wait. The client MAY repeat the request without modifications at any later time.
```
:::

Check out [fauxpas](https://github.com/ropenscilabs/fauxpas) for more
information about HTTP exceptions.

## Allowing real requests

You can always disable `webmockr` by using `webmockr::disable()`, which
completelydisables mocking.

You can also allow some real requests while still using `webmockr`.

One way to allow some requests is to execute
`webmockr_allow_net_connect()`, whichdoesn't disable `webmockr`
completely - it allows real HTTP requests to bemade that have no stubs
associated.

Another way is to disallow all requests except for certain URIs
using`webmockr_disable_net_connect()`. For example, if we
run`webmockr_disable_net_connect("google.com")` no real HTTP requests
are allowedother than those that match `"google.com"`.

You can also allow only localhost HTTP requests with the
`allow_localhost` parameterin the `webmockr_configure()` function. Run
`webmockr_configure(allow_localhost = TRUE)`,and then all localhost
requests will be allowed while all non-locahostrequests will not be
allowed. `allow_localhost` works for all the localhost
variants:`localhost`, `127.0.0.1`, and `0.0.0.0`.

You can check whether you are allowing real requests
with`webmockr_net_connect_allowed()`, and you can see your `webmockr`
configuration with`webmockr_configuration()`.

## Storing actual HTTP responses

`webmockr` doesn't do that. Check out
[vcr](https://github.com/ropensci/vcr) for that.

Some have
[noted](https://quickleft.com/blog/decreasing-vcr-dependency-webmock/)
that in some cases storing actual http responses can get to be too
cumbersome (maybe too much disk space, or other reasons) - and have
reverted from using a tool like `vcr` that caches real responses to
using something like `webmockr` that only "stubs" fake responses.

There's many options for testing a library that does HTTP requests:

1.  Do actual HTTP requests
2.  Stub HTTP requests with a tool like `webmockr`
3.  Cache real HTTP responses with a too like `vcr`

## webmockr for your test suite

You can use `webmockr` for your test suite right now. Here's a quick
example of using`webmockr` with `testthat`

::: {.highlight}
``` {.chroma tabindex="0"}
library(crul)library(testthat)
```
:::

Make a stub

::: {.highlight}
``` {.chroma tabindex="0"}
stub_request("get", "https://httpbin.org/get") %>%   to_return(body = "success!", status = 200)#> <webmockr stub> #>   method: get#>   uri: https://httpbin.org/get#>   with: #>     query: #>     body: #>     request_headers: #>   to_return: #>     status: 200#>     body: success!#>     response_headers: #>   should_timeout: FALSE#>   should_raise: FALSE
```
:::

Check that it's in the stub registry

::: {.highlight}
``` {.chroma tabindex="0"}
stub_registry()#> <webmockr stub registry> #>  Registered Stubs#>    get: https://httpbin.org/get   | to_return:  with body "success!"  with status 200
```
:::

Make the request

::: {.highlight}
``` {.chroma tabindex="0"}
z <- crul::HttpClient$new(url = "https://httpbin.org")$get("get")
```
:::

Run tests (nothing returned means it passed)

::: {.highlight}
``` {.chroma tabindex="0"}
expect_is(z, "HttpResponse")expect_equal(z$status_code, 200)expect_equal(z$parse("UTF-8"), "success!")
```
:::

## Todo

There are a number of things to still get done with `webmockr`

- Integrating into `curl` and `httr` for mocking requests with those
  libraries.
- Support [multiple responses for repeated
  requests](https://github.com/bblimke/webmock#multiple-responses-for-repeated-requests)
- Allow to [specify number of times a given response should be
  returned](https://github.com/ropensci/webmockr/issues/10), [see
  also](https://github.com/bblimke/webmock#specifying-number-of-times-given-response-should-be-returned)
- Support regex patterns for URI's and [URI
  templates](https://github.com/ropensci/webmockr/issues/20)
- and probably lots more ...

## Feedback!

We'd love to get some eyes on this; to sort out problems that will no
doubt arise from real world scenarios; to flesh out new use cases we
hadn't thought of, etc.

Open an issue: <https://github.com/ropensci/webmockr/issues/new>


webmockr  webmockr is an R library for stubbing and setting expectations on HTTP requests.It is a port of the Ruby gem webmock. webmockr works by plugging in to another R package that does HTTP requests. It currently only works with crul right now, but we plan to add support for curl and httr later.

Postagens de Rogue Scholar

We cleaned our website URLs with R!

webmockr: mock HTTP requests