Maëlle Salmon

As part of our work [documenting
R-Universe](/blog/2024/04/12/gsod-announcement/),we're adding
screenshots of the interface to the [documentation
website](https://docs.r-universe.org).Taking screenshots manually could
quickly become very cumbersome, especially as we expect they'll need
updating in future: we might want to change the universes we feature,
the interface might improve [yet
again](/blog/2024/06/12/runiverse-frontend/) and therefore look slightly
different.Therefore, we decided to opt for a programmatic approach.In
this post we shall present our learnings from using the R packages
[chromote](https://rstudio.github.io/chromote/) and
[magick](https://docs.ropensci.org/magick/) to produce screenshots.

## Main packages needed

### The chromote R package

To take a screenshot programmatically, we need to somehow get control of
a browser from a script.The [chromote R
package](https://rstudio.github.io/chromote/) is an actively maintained
wrapper for [Chrome Remote
Interface](https://github.com/cyrus-and/chrome-remote-interface),
authored by Winston Chang and Barrett Schloerke.With chromote, you can
open a browser in the background, navigate to the page of your choice,
interact with it, and capture screenshots.chromote powers the
experimental [live web-scraping in the rvest
package](https://rvest.tidyverse.org/reference/read_html_live.html) and
is also used in [shinytest2](https://rstudio.github.io/shinytest2/).

### The magick R package

Generally, [Jeroen Ooms'](/author/jeroen-ooms) [magick R
package](https://docs.ropensci.org/magick/) isn't required to take
screenshots, but we wanted to format the images by adding
*shadows*.Therefore this tool for image manipulation was necessary.

## A first screenshot

We start by initiating a chromote session of a standard screen size.We
found the default screen size too narrow for our needs.

::: {.highlight}
``` {.chroma tabindex="0"}
library("chromote")screen_width <- 1920screen_height <- 1080b <- ChromoteSession$new(height = screen_height, width = screen_width)
```
:::

We created a function that wraps `b$screenshot()` while still
exposingthe parameters we need to tweak for some of the pages.The
chromote package, like Shiny, uses R6.

Our function also calls the handy
[`magick::image_shadow()`](https://docs.ropensci.org/magick/reference/composite.html)
for adding a shadow to the image.

::: {.highlight}
``` {.chroma tabindex="0"}
screenshot <- function(b, img_path,                       selector = "html",                       cliprect = c(top = 0, left = 0, width = 1920, height = 1080),                       expand = NULL) {  b$screenshot(    img_path,    selector = selector,    delay = 2,    cliprect = cliprect,    expand = expand  )  magick::image_read(img_path) |>    magick::image_shadow() |>    magick::image_write(img_path, quality = 100)}
```
:::

We can now capture the search interface of R-Universe.In our script,
we've added a generous `Sys.sleep()` call to ensure pages are properly
loaded.

::: {.highlight}
``` {.chroma tabindex="0"}
b$Page$navigate("https://r-universe.dev/search/")Sys.sleep(1)screenshot(b, "search.png")
```
:::

This example captures the screenshot below.

<figure>
<img src="/blog/2024/09/10/script-screenshots/search.png"
alt="Screenshot of R-universe search interface, showing the search bar" />
</figure>

## Screenshots after filling a text input area

We also wanted to demonstrate searching for something.We followed the
example in a
[gist](https://gist.github.com/oganm/50a8020f718842aa3eee04dcfd57c198)
to enter text into the search navbar^[1](#fn:1){.footnote-ref
role="doc-noteref"}^:

- we needed to find the search input area, whose idea is "search-input",
  by a call to `b$DOM$querySelector()`;
- then we focussed on that area through `b$DOM$focus()`;
- finally we added text with `b$Input$insertText()`.

It was easier to reload the page than to try and delete the current text
before doing a new search.

::: {.highlight}
``` {.chroma tabindex="0"}
screenshot_search <- function(query, screen_width) {  b$Page$navigate("https://r-universe.dev/search/")  Sys.sleep(2)  search_box <- b$DOM$querySelector(b$DOM$getDocument()$root$nodeId, "#search-input")  b$DOM$focus(search_box$nodeId)  b$Input$insertText(text = query)  Sys.sleep(2)  screenshot(    b, sprintf("search-%s.png", snakecase::to_lower_camel_case(query))  )}purrr::walk(  c('"missing-data"', "author:jeroen json", "exports:tojson"),  screenshot_search,  screen_width = screen_width)
```
:::

## Screenshots after clicking on something

The search interface of R-universe suggests some advanced research
fields if one clicks on the arrow-down button near the search
button.Through a GitHub search for code we found an example of clicking
in the [source code of
rvest](https://github.com/tidyverse/rvest/blob/c9be5b8dd9d672e84dd0dc515e3a37ab5c03111f/R/live.R#L145)
that we were able to adapt.

The code finds the arrow-down button through its class rather than id so
it might be a bit brittle over time.Once the class is found, we retrieve
its "box model" which is a bunch of coordinates, and then calculate the
centre coordinates.After that, a "click" which is actually a three-step
action: moving the mouse, pressing the mouse, releasing the mouse.

::: {.highlight}
``` {.chroma tabindex="0"}
# Searching, advanced fields ----b$Page$navigate("https://r-universe.dev/search/")Sys.sleep(1)search_info <- b$DOM$querySelector(  b$DOM$getDocument()$root$nodeId,  "button.btn.btn-outline-secondary.dropdown-toggle.dropdown-toggle-split")quads <- b$DOM$getBoxModel(search_info$nodeId)content_quad <- as.numeric(quads$model$content)center_x <- mean(content_quad[c(1, 3, 5, 7)])center_y <- mean(content_quad[c(2, 4, 6, 8)])b$Input$dispatchMouseEvent(  type = "mouseMoved",  x = center_x,  y = center_y,)b$Input$dispatchMouseEvent(  type = "mousePressed",  x = center_x,  y = center_y,  button = "left",  clickCount = 1)b$Input$dispatchMouseEvent(  type = "mouseReleased",  x = center_x,  y = center_y,  button = "left",  clickCount = 1)Sys.sleep(2)screenshot(b, "search-advanced.png")
```
:::

<figure>
<img src="/blog/2024/09/10/script-screenshots/search-advanced.png"
alt="Screenshot of R-universe search interface, showing the search bar and the advanced search fields" />
</figure>

## Other gotchas about using chromote

The package pages on R-universe are quite exhaustive.We needed to
capture screenshots of different sections.When using an URL with a
fragment, for instance `https://r-spatial.r-universe.dev/sf#citation`,we
got the exact same screenshot as for
`https://r-spatial.r-universe.dev/sf`.

We therefore had to use the `cliprect` argument of the screenshot
method.Figuring out we needed it was a first gotcha.

A second gotcha is that `cliprect` is ["a (unnamed) vector/list with
left, top, width and
height"](https://github.com/rstudio/chromote/issues/168#issuecomment-2302422282)^[2](#fn:2){.footnote-ref
role="doc-noteref"}^:and it's the *order*, not the names, that matters.

## Conclusion

In this post we explained how we used the chromote and magick R packages
to produce screenshots for the documentation website of R-universe.Find
the [current version of our
script](https://github.com/r-universe-org/hitchhikers-guide/blob/main/R/screenshots.R).Do
you use sometimes chromote or a similar programmatic browser interface?

::: {.footnotes role="doc-endnotes"}

------------------------------------------------------------------------

1.  ::: {#fn:1}
    Our need to interact with the webpage *before* capturing the
    screenshot prevented our using
    [webshot2](https://rstudio.github.io/webshot2/). [↩︎](#fnref:1){.footnote-backref
    role="doc-backlink"}
    :::

2.  ::: {#fn:2}
    We appreciated the fast and helpful user
    support! [↩︎](#fnref:2){.footnote-backref role="doc-backlink"}
    :::
:::


As part of our work documenting R-Universe,we’re adding screenshots of the interface to the documentation website.Taking screenshots manually could quickly become very cumbersome, especially as we expect they’ll need updating in future: we might want to change the universes we feature, the interface might improve yet again and therefore look slightly different.Therefore, we decided to opt for a programmatic approach.In this post we shall present

Capturing Screenshots Programmatically With R

Mauro Lepore

Recently [Yanina Bellini Saibene](/author/yanina-bellini-saibene/)
reminded usto update our Slack profile:

> Friendly reminder: Let's increase the value of our rOpenSci Slack
> community.Please add details to your profile, e.g., your photo, your
> favorite social mediahandle, what you do, your pronouns, and how to
> pronounce your name.

After doing that I went on to updating my profile photos on the
rOpenSciwebsite, which ended up teaching me a few git tricks I would
like to share here.Thanks [Maëlle Salmon](/author/ma%C3%ABlle-salmon/)
for the encouragement, and[Steffi LaZerte](/author/steffi-lazerte/) for
reviewing this post.

## Cloning as usual

When I tried to clone the source code of rOpenSci's website I realized
the repowas large and it would take me several minutes.

::: {.highlight}
``` {.chroma tabindex="0"}
git clone https://github.com/ropensci/roweb3.git
```
:::

I decided to stop the process and researched how to just pull the latest
versionof the specific files I needed.

## Pulling the latest version of specific files

First I forked the rOpenSci website repository (`roweb3`). I used
the[`gh`](https://cli.github.com/) CLI from the terminal, but also I
could haveforked it manually from Github.

::: {.highlight}
``` {.chroma tabindex="0"}
# if not using `gh`, fork ropensci/roweb3 from GitHubgh repo fork ropensci/roweb3
```
:::

Then I created a local empty `roweb3` directory and linked it to the
fork.

::: {.highlight}
``` {.chroma tabindex="0"}
git init roweb3cd roweb3git remote add origin git@github.com:maurolepore/roweb3.git
```
:::

Now for the tricks! I avoided having to download the whole repository by
firstfinding the specific files I needed on GitHub's "Go to file" box,
then:

- Trick 1: Configured a[sparse
  checkout](https://git-scm.com/docs/git-sparse-checkout) matching
  justthose files.

::: {.highlight}
``` {.chroma tabindex="0"}
git config core.sparseCheckout trueecho "themes/ropensci/static/img/team/mauro*" >> .git/info/sparse-checkout
```
:::

- Trick 2: Pulled
  with[`--depth 1`](https://git-scm.com/docs/git-pull#Documentation/git-pull.txt---depthltdepthgt)to
  get only the latest version of those files.

::: {.highlight}
``` {.chroma tabindex="0"}
git pull --depth=1 origin main
```
:::

I explored the result
with[`tree`](https://manpages.ubuntu.com/manpages/bionic/man1/tree.1.html)
and itwas just what I needed to modify:

::: {.highlight}
``` {.chroma tabindex="0"}
tree.└── themes    └── ropensci        └── static            └── img                └── team                    ├── mauro-lepore.jpg                    └── mauro-lepore-mentor.jpg
```
:::

## But how large is it?

While those tricks were useful, I was still curious about the size of
the repo,so I did clone it all and explored disk usage
with[`du`](https://manpages.ubuntu.com/manpages/bionic/man1/du.1.html):

::: {.highlight}
``` {.chroma tabindex="0"}
du --human-readable --max-depth=1 .219M    ./themes164K    ./.Rproj.user56K     ./archetypes628K    ./resources168K    ./data376M    ./.git20K     ./static12K     ./.github40K     ./scripts161M    ./content24K     ./layouts475M    ./public1.3G    .
```
:::

Indeed this is much larger than the source code I typically handle. But
now Iknow a few more Git tricks (and even more about blogging on
rOpenSci :-) ).

## Conclusion

> If all you have is a hammer, everything looks like a nail. --- Abraham
> Maslow

Sometimes `git clone` is not the right tool for the job. A sparse
checkout and ashallow pull can help you get just what you need.

If you enjoy learning from videos you may [search "git" on my
YouTubechannel](https://www.youtube.com/leporemauro/search?query=git) or
explore
theplaylists[git](https://www.youtube.com/playlist?list=PLvgdJdJDL-AOHkwiaMvYhPKVjiD9vzZIo),[git-from-the-terminal](https://www.youtube.com/playlist?list=PLvgdJdJDL-AMyv06bsXoXkGmxmaV9U6Ts),and[git-con-la-terminal](https://www.youtube.com/playlist?list=PLvgdJdJDL-APwLSt89PJgI72UGVNUjOKl)(in
Spanish).

What are your favorite Git tricks? How about blogging about them?


Recently Yanina Bellini Saibene reminded usto update our Slack profile: After doing that I went on to updating my profile photos on the rOpenSciwebsite, which ended up teaching me a few git tricks I would like to share here.Thanks Maëlle Salmon for the encouragement, andSteffi LaZerte for reviewing this post.Cloning as usual  When I tried to clone the source code of rOpenSci’s website I realized the repowas large and it would take me several

Git Tricks for Working with Large Repositories

rOpenSci's [second cohort of Champions has been
onboarded](/blog/2024/02/15/champions-program-champions-2024/)!Their
training first started with a [session on code
style](/blog/2024/02/22/beautiful-code/), was followed by three sessions
on the basics of R package development, and ended with a session on
[advanced R package
development](https://rpkgdev-rocket-2024.netlify.app/), which consisted
of a potpourri of tips with discussion, followed by time for applying
these principles to the participants' packages.Here, I want to share one
of the topics covered: Package testing, and in particular, the DRY
("don't repeat yourself") and DAMP ("descriptive and meaningful
phrases") principles.For this topic, we used a [GitHub
repository](https://github.com/maelle/swamp), containing an R package
whose different commits illustrate the two principles. In each step
we'll share a commit or diff illustrating the changes made.

## Stage 1: muddy!

First commit: Set up our test files,
[test-works.R](https://github.com/maelle/swamp/blob/c67cc053cafb2cc5f5125cbc745b2f392a84e3df/tests/testthat/test-works.R)
and
[test-ok.R](https://github.com/maelle/swamp/blob/c67cc053cafb2cc5f5125cbc745b2f392a84e3df/tests/testthat/test-ok.R)

These initial test files both define an object called `test_object` at
the top level, which is used in two tests in each file.

**test-works.R**

::: {.highlight}
``` {.chroma tabindex="0"}
test_object <- list(a = 1, b = 2)test_that("multiplication works", {  expect_equal(test_object[["b"]] * 2, 4)})test_that("addition works", {  expect_equal(test_object[["a"]] + 2, 3)})
```
:::

and

**test-ok.R**

::: {.highlight}
``` {.chroma tabindex="0"}
test_object <- list(a = 1, b = 2)test_that("division works", {  expect_equal(test_object[["b"]] / 2, 1)})test_that("substraction works", {  expect_equal(test_object[["a"]] - 1, 0)})
```
:::

It is not an optimal pattern because you can't look at each
`test_that()` test in isolation and quickly understand what is going
on.In a very long test file you would even have to scroll up and down!
😱Furthermore we are being repetitive by defining the same test object
in two test files.

## Stage 2: DRY!

Next commit: [DRY out these
files](https://github.com/maelle/swamp/commit/381f244f56f1837207f2150a7e76c70bd59c0422)

At this stage we diligently remember about DRY, Don't Repeat Yourself,
and about the mechanics of [testthat helper
files](https://blog.r-hub.io/2020/11/18/testthat-utility-belt/):files
whose names start with `helper-` are loaded before all tests.

So we create a helper file (`helper-swamp.R`) within which `test_object`
is defined and therefore available for tests!

In `tests/testthat/helper-swamp.R`,

::: {.highlight}
``` {.chroma tabindex="0"}
test_object <- list(a = 1, b = 2)
```
:::

In the test files, we removed the first line that defined `test_object`.

Now things aren't perfect yet.When we look at any of the test files, we
can't really know what `test_object` is as its name is not "descriptive
and meaningful".

Furthermore we now have `test_object` which is *always* defined even if
it isn't used in a test.At best this is unnecessary and useless; at
worst it could have unintended side-effects, particularly with more
complex code! ^[1](#fn:1){.footnote-ref role="doc-noteref"}^

## Stage 3: Focus on DAMP

Third commit: [Apply DAMP
principals](https://github.com/maelle/swamp/commit/9cc08937e1cabf112955004212e1d0bccf2ccda4)

In the helper file, `tests/testthat/helper-swamp.R`, we re-factor the
code into a *function* with a more meaningful name (at least let's
pretend it is!).

::: {.highlight}
``` {.chroma tabindex="0"}
basic_list <- function() {  list(a = 1, b = 2)}
```
:::

Next, we call this function to define the object in all tests where it's
needed.Thus the test files become

::: {.highlight}
``` {.chroma tabindex="0"}
test_that("division works", {  test_object <- basic_list()  expect_equal(test_object[["b"]] / 2, 1)})
```
:::

and

::: {.highlight}
``` {.chroma tabindex="0"}
test_that("substraction works", {  test_object <- basic_list()  expect_equal(test_object[["a"]] - 1, 0)})
```
:::

Now, while the actual definition of the basic list is not in all tests,
we have a better idea of what's going on when reading the test.

Further, if the test were to fail, in the console we could run
`devtools::load_all()` and run the code of the test, as
`devtools::load_all()` loads testthat helper files so `basic_list()`
would be available.

## Conclusion

The balance between DRY ("Don't repeat yourself") and DAMP ("Descriptive
and meaningful phrases") is a trade-off.To keep the water analogy, we
also need to ensure that our code doesn't have effects which may 'leak'
unexpectedly.What we should strive for are self-contained tests that we
can understand and run without too much context.

Another consideration we haven't covered here, are tests that require
specific elements, such as environment variables or options. In those
cases try using [withr](https://withr.r-lib.org/) functions such as
[`withr::local_envvar()`](https://withr.r-lib.org/reference/with_envvar.html)
in each test that requires it.

One powerful idea from the book ["Software Engineering at
Google"](https://www.oreilly.com/library/view/software-engineering-at/9781492082781/)
by Titus Winters, Tom Manshreck, and Hyrum Wright, is that code can
afford to be a bit less obvious because it has tests covering it, but
test code, which isn't covered by tests, does not have this luxury.

## Further resources

- The three chapters on package testing of the R packages book by Hadley
  Wickham and Jenny Bryan are a highly recommended read: [Testing
  basics](https://r-pkgs.org/testing-basics.html), [Designing your test
  suite](https://r-pkgs.org/testing-design.html), [Advanced testing
  techniques](https://r-pkgs.org/testing-advanced.html).

- Blog post [Why Good Developers Write Bad Unit
  Tests](https://mtlynch.io/good-developers-bad-tests/) by Michael
  Lynch.

::: {.footnotes role="doc-endnotes"}

------------------------------------------------------------------------

1.  ::: {#fn:1}
    This is a leak. In another test, one could wonder why some object
    exists, why a specific option has been set, etc. and it's a bit of a
    nightmare to debug. [↩︎](#fnref:1){.footnote-backref
    role="doc-backlink"}
    :::
:::


rOpenSci’s second cohort of Champions has been onboarded!Their training first started with a session on code style, was followed by three sessions on the basics of R package development, and ended with a session on advanced R package development, which consisted of a potpourri of tips with discussion, followed by time for applying these principles to the participants’ packages.Here, I want to share one of the topics covered: Package testing, and

Rogue Scholar Beiträge

Capturing Screenshots Programmatically With R

Git Tricks for Working with Large Repositories

An Example of the DRY/DAMP Principles for Package Tests