Will Landau

Make
<sup>
 1
</sup>
-like pipelines enhance the integrity, transparency, shelf life, efficiency, and scale of large analysis projects.With pipelines, data science feels smoother and more rewarding, and the results are worthy of more trust. targets  install.packages("targets")  The targets
<sup>
 2
</sup>
package is a new pipeline toolkit for R.It recently cleared software review, and it is now on CRAN.

targets: Democratizing Reproducible Analysis Pipelines

The [`drake` R package](https://github.com/ropensci/drake) is not only a
reproducible research solution, but also a serious high-performance
computing engine. The [package
website](https://docs.ropensci.org/drake/) introduces
[`drake`](https://github.com/ropensci/drake), and this technical note
draws from the guides on high-performance computing and timing in the
[`drake` manual](https://github.com/ropenscilabs/drake-manual).

### You can help!

Some of these features are brand new, and others are newly refactored.
The [GitHub version](https://github.com/ropensci/drake) has all the
advertised functionality, but it needs more testing and development
before I can submit it to CRAN in good conscience. New issues such as
[r-lib/processx#113](https://github.com/r-lib/processx/issues/113) and
[HenrikBengtsson/future#226](https://github.com/HenrikBengtsson/future/issues/226)
seem to affect [`drake`](https://github.com/ropensci/drake), and more
may emerge. If you use [`drake`](https://github.com/ropensci/drake) for
your own work, please consider supporting the project by field-testing
the claims below and posting feedback
[here](https://github.com/ropensci/drake/issues/369).

### Let drake schedule your targets.

A typical workflow is a sequence of interdependent data transformations.

When you call `make()` on this project,
[`drake`](https://github.com/ropensci/drake) takes care of
`"raw_data.xlsx"`, then `raw_data`, and then `data` in sequence. Once
`data` completes, `fit` and `hist` can launch in parallel, and then
`"report.md"` begins once everything else is done. It is
[`drake`](https://github.com/ropensci/drake)'s responsibility to deduce
this order of execution, hunt for ways to parallelize your work, and
free you up to focus on the substance of your research.

### Activate parallel processing.

Simply set the `jobs` argument to an integer greater than 1. The
following `make()` recruits multiple processes on your local machine.

::: {.highlight}
``` {.chroma tabindex="0"}
make(plan, jobs = 2)
```
:::

For parallel deployment to a computing cluster
([SLURM](https://slurm.schedmd.com/),
[TORQUE](https://www.adaptivecomputing.com/products/open-source/torque/),
[SGE](http://www.univa.com/products/), etc.)
[`drake`](https://github.com/ropensci/drake) calls on packages
[`future`](https://github.com/HenrikBengtsson/future),
[`batchtools`](https://github.com/mllg/batchtools), and
[`future.batchtools`](https://github.com/HenrikBengtsson/future.batchtools).
First, create a [`batchtools` template
file](https://github.com/mllg/batchtools/tree/master/inst/templates) to
declare your resource requirements and [environment
modules](http://modules.sourceforge.net/). There are built-in example
files in [`drake`](https://github.com/ropensci/drake), but you will
likely need to tweak your own by hand.

::: {.highlight}
``` {.chroma tabindex="0"}
drake_batchtools_tmpl_file("slurm") # Writes batchtools.slurm.tmpl.
```
:::

Next, tell
[`future.batchtools`](https://github.com/HenrikBengtsson/future.batchtools)
to talk to the cluster.

::: {.highlight}
``` {.chroma tabindex="0"}
library(future.batchtools)future::plan(batchtools_slurm, template = "batchtools.slurm.tmpl")
```
:::

Finally, set `make()`'s `parallelism` argument equal to `"future"` or
`"future_lapply"`.

::: {.highlight}
``` {.chroma tabindex="0"}
make(plan, parallelism = "future", jobs = 8)
```
:::

### Choose a scheduling algorithm.

The `parallelism` argument of `make()` controls not only where to deploy
the workers, but also how to schedule them. The following table
categorizes the 7 options.

<table>
<colgroup>
<col style="width: 33%" />
<col style="width: 33%" />
<col style="width: 33%" />
</colgroup>
<thead>
<tr class="header">
<th></th>
<th style="text-align: center;">Deploy: local</th>
<th style="text-align: right;">Deploy: remote</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>Schedule: persistent</strong></td>
<td style="text-align: center;">“mclapply”, “parLapply”</td>
<td style="text-align: right;">“future_lapply”</td>
</tr>
<tr class="even">
<td><strong>Schedule: transient</strong></td>
<td style="text-align: center;"></td>
<td style="text-align: right;">“future”, “Makefile”</td>
</tr>
<tr class="odd">
<td><strong>Schedule: staged</strong></td>
<td style="text-align: center;">“mclapply_staged”,
“parLapply_staged”</td>
<td style="text-align: right;"></td>
</tr>
<tr class="even">
<td><br />
</td>
<td style="text-align: center;"></td>
<td style="text-align: right;"></td>
</tr>
</tbody>
</table>

#### Staged scheduling

[`drake`](https://github.com/ropensci/drake)'s first custom parallel
algorithm was staged scheduling. It was easier to implement than the
other two, but the workers run in lockstep. In other words, all the
workers pick up their targets at the same time, and each worker has to
finish its target before any worker can move on. The following animation
illustrates the concept.

::: {.wistia_responsive_padding style="padding:56.21% 0 0 0;position:relative;"}
::: {.wistia_responsive_wrapper style="height:100%;left:0;position:absolute;top:0;width:100%;"}
::: {.wistia_embed .wistia_async_uxzk0qgy9e .videoFoam=true style="height:100%;position:relative;width:100%"}
::: {.wistia_swatch style="height:100%;left:0;opacity:0;overflow:hidden;position:absolute;top:0;transition:opacity 200ms;width:100%;"}
![](https://fast.wistia.com/embed/medias/uxzk0qgy9e/swatch){style="filter:blur(5px);height:100%;object-fit:contain;width:100%;"
onload="this.parentNode.style.opacity=1;"}
:::
:::
:::
:::

But despite weak parallel efficiency, staged scheduling remains useful
because of its low overhead. Without the bottleneck of a formal master
process, staged scheduling blasts through armies of tiny conditionally
independent targets. Consider it if the bulk of your work is finely
diced and [perfectly
parallel](https://en.wikipedia.org/wiki/Embarrassingly_parallel), maybe
if your dependency graph is tall and thin.

    library(dplyr)library(drake)N <- 500gen_data <- function() {  tibble(a = seq_len(N), b = 1, c = 2, d = 3)}plan_data <- drake_plan(  data = gen_data())plan_sub <-  gen_data() %>%  transmute(    target = paste0("data", a),    command = paste0("data[", a, ", ]")  )plan <- bind_rows(plan_data, plan_sub)plan## # A tibble: 501 x 2##    target command   ##           ##  1 data   gen_data()##  2 data1  data[1, ] ##  3 data2  data[2, ] ##  4 data3  data[3, ] ##  5 data4  data[4, ] ##  6 data5  data[5, ] ##  7 data6  data[6, ] ##  8 data7  data[7, ] ##  9 data8  data[8, ] ## 10 data9  data[9, ] ## # ... with 491 more rowsconfig <- drake_config(plan)vis_drake_graph(config)

::: {style="text-align: center"}
:::

#### Persistent scheduling

Persistent scheduling is brand new to
[`drake`](https://github.com/ropensci/drake). Here, `make(jobs = 2)`
deploys three processes: two workers and one master. Whenever a worker
is idle, the master assigns it the next target whose dependencies are
fully ready. The workers keep running until no more targets remain. See
the animation below.

::: {.wistia_responsive_padding style="padding:56.21% 0 0 0;position:relative;"}
::: {.wistia_responsive_wrapper style="height:100%;left:0;position:absolute;top:0;width:100%;"}
::: {.wistia_embed .wistia_async_ycczhxwkjw .videoFoam=true style="height:100%;position:relative;width:100%"}
::: {.wistia_swatch style="height:100%;left:0;opacity:0;overflow:hidden;position:absolute;top:0;transition:opacity 200ms;width:100%;"}
![](https://fast.wistia.com/embed/medias/ycczhxwkjw/swatch){style="filter:blur(5px);height:100%;object-fit:contain;width:100%;"
onload="this.parentNode.style.opacity=1;"}
:::
:::
:::
:::

#### Transient scheduling

If the time limits of your cluster are too strict for persistent
workers, consider transient scheduling, another new arrival. Here,
`make(jobs = 2)` starts a brand new worker for each individual target.
See the following video.

::: {.wistia_responsive_padding style="padding:56.21% 0 0 0;position:relative;"}
::: {.wistia_responsive_wrapper style="height:100%;left:0;position:absolute;top:0;width:100%;"}
::: {.wistia_embed .wistia_async_340yvlp515 .videoFoam=true style="height:100%;position:relative;width:100%"}
::: {.wistia_swatch style="height:100%;left:0;opacity:0;overflow:hidden;position:absolute;top:0;transition:opacity 200ms;width:100%;"}
![](https://fast.wistia.com/embed/medias/340yvlp515/swatch){style="filter:blur(5px);height:100%;object-fit:contain;width:100%;"
onload="this.parentNode.style.opacity=1;"}
:::
:::
:::
:::

  

### How many jobs should you choose?

The `predict_runtime()` function can help. Let's revisit the [`mtcars`
example](https://docs.ropensci.org/drake/reference/load_mtcars_example.html).

::: {style="text-align: center"}
:::

Let's also

1.  Plan for non-staged scheduling,
2.  Assume each non-file target (black circle) takes 2 hours to build,
    and
3.  Rest assured that everything else is super quick.

When we declare the runtime assumptions with the `known_times` argument
and cycle over a reasonable range of `jobs`, `predict_runtime()` paints
a clear picture.

<figure class="center">
<img
src="https://cdn.rawgit.com/ropensci/drake/21c41083/images/times.png"
width="600" />
</figure>

`jobs = 4` is a solid choice. Any fewer would slow us down, and the next
2-hour speedup would take double the `jobs` and the hardware to back it
up. Your choice of `jobs` for `make()` ultimately depends on the runtime
you can tolerate and the computing resources at your disposal.

### Thanks!

When I attended
[`RStudio::conf(2018)`](https://www.rstudio.com/conference/),
[`drake`](https://github.com/ropensci/drake) relied almost exclusively
on staged scheduling. [Kirill Müller](https://github.com/krlmlr) spent
hours on site and hours afterwards helping me approach the problem and
educating me on [priority
queues](https://en.wikipedia.org/wiki/Priority_queue), [message
queues](https://en.wikipedia.org/wiki/Message_queue), and the [knapsack
problem](https://en.wikipedia.org/wiki/Knapsack_problem). His generous
help paved the way for [`drake`](https://github.com/ropensci/drake)'s
latest enhancements.

### Disclaimer

This post is a product of my own personal experiences and opinions and
does not necessarily represent the official views of my employer. I
created and embedded the [Powtoon](https://www.powtoon.com) videos only
as explicitly permitted in the [Terms and Conditions of
Use](https://www.powtoon.com/terms-and-conditions/), and I make no
copyright claim to any of the constituent graphics.


The drake R package is not only a reproducible research solution, but also a serious high-performance computing engine. The package website introduces drake, and this technical note draws from the guides on high-performance computing and timing in the drake manual. You can help! Some of these features are brand new, and others are newly refactored.

Messaggi di Rogue Scholar

targets: Democratizing Reproducible Analysis Pipelines

drake's improved high-performance computing power