Jeroen Ooms

## Downloading repository snapshots

The new snapshot API lets you download a full copy of any CRAN-like
repository on r-universe. You can use such a snapshot to mirror the
entire CRAN-like repository on your own servers, or for example to build
a stable, validated release of your package suite.

The API endpoint is simply `/api/snapshot` and has several options to
filter content. By default it returns a zip file with all the packages,
binaries, and documentation in your repository. If this is more than you
need, there are parameters to include e.g. only binaries for certain
platforms or certain versions of R, or to create a repository from a
subset of the packages in your universe. Explore the parameters on the
API tab of any universe, for example: <https://ggseg.r-universe.dev/api>

::: {.box}
<figure itemprop="associatedMedia" itemscope=""
itemtype="http://schema.org/ImageObject">
<div class="img">
<img src="screenshot.png" itemprop="thumbnail"
alt="Screenshot of r-universe API explorer tab." />
</div>
<a href="screenshot.png" itemprop="contentUrl"></a>
</figure>
:::

## How to use snapshots

A CRAN-like R package repository is essentially a static directory of
package files and indexes, with a specific naming structure. R-universe
automatically builds and updates all these files based on a registry of
packages and upstream git sources. But once everything is generated, you
can simply copy the entire folder elsewhere, and have a frozen CRAN-like
package repository on that server!

For example, you could create a github-action which regularly downloads
a snapshot from r-universe to publish on github-pages. Here is a minimal
example of such an action: <https://github.com/jeroen/backup>. Note how
the action is vary fast: downloading and extracting te snapshot from
r-universe only takes a few seconds. So this is easily something that
could be done on demand, or automatically on a regular basis.

You can also install packages in R directly from a local snapshot
folder, by prefixing the path that you pass to `install.packages` with
`file://` (Windows paths need to be normalized to look more unixy):

::: {.highlight}
``` {.chroma tabindex="0"}
# Download and extract the snapshotcurl::curl_download("https://jeroen.r-universe.dev/api/snapshot/zip?binaries=4.3", "snapshot.zip")snapshot <- file.path(tempdir(), 'jeroen')unzip("snapshot.zip", exdir = snapshot)# Install packages from the local repositoryprefix <- ifelse (.Platform$OS.type == "windows", "file:///", "file://")repos <- paste0(prefix, normalizePath(snapshot, "/"))install.packages(c("V8", "mongolite"), repos = repos)
```
:::

## Towards 'validated' stable R packages repositories

Snapshots can be useful for several purposes, e.g for archiving, or to
mirror an R package repository within a corporate intranet. But there is
one important use case we want to highlight:

One gap in the R ecosystem [has always
been](https://journal.r-project.org/archive/2013-1/ooms.pdf) the lack of
staged, stable package repositories. Packages on CRAN and R-universe and
other networks^[1](#fn:1){.footnote-ref role="doc-noteref"}^ get
constantly updated, pushing major changes directly to end-users. This is
convenient for developers but not ideal for production and
reproducibility.

By comparison, Linux distributions (like Ubuntu, RHEL) also consist of a
large set of constantly changing software packages. However most Linux
end-users do not get updates directly, because their system gets updates
from a stable branch of the package repositories. Such a stable branch
is like a snapshot of the entire package repository, which only gets
small critical updates; never major changes. This way end-users are
shielded from unexpected big changes until they decide to upgrade to a
new release of the Linux distribution, which basically just changes the
package repository to the next stable branch.

## Using registries and snapshots for controlled repositories

By default, packages in r-universe get updated from their git
master/main branch. However all tools are available for controlling the
R package repository to build and publish a custom validated set R
packages.

In the `packages.json` registry file where you list the git-urls to
include, you can [specify which branch or tag to
track](https://ropensci.org/blog/2021/06/22/setup-runiverse/#pro-tip-tracking-custom-branches-or-releases)
for all packages in your universe. You can even set this value to a
fixed commit. This way you control exactly which version of each R
package gets included in your cranlike repository, such that you can
make sure that version passes your criteria or review process.

Once your universe contains the packages and versions that you want (say
for use by a given paper or application), you can use the snapshot API
to create a release of your cranlike repository. By hosting or sharing
this snapshot, you provide permanent easy access to the frozen cranlike
package repository with exactly the packages and versions that you
validated or can use to (re)produce particular results.

::: {.footnotes role="doc-endnotes"}

------------------------------------------------------------------------

1.  ::: {#fn:1}
    One notable exception is bioconductor, which does have stable
    releases of the entire bioconductor package
    network. [↩︎](#fnref:1){.footnote-backref role="doc-backlink"}
    :::
:::


Downloading repository snapshots  The new snapshot API lets you download a full copy of any CRAN-like repository on r-universe. You can use such a snapshot to mirror the entire CRAN-like repository on your own servers, or for example to build a stable, validated release of your package suite. The API endpoint is simply /api/snapshot and has several options to filter content.

Downloading snapshots and creating stable R packages repositories using r-universe

## How packages appear in r-universe

Last month we explained how r-universe makes it easy to search and
browse through the countless R packages, articles, and datasets to let
you [discover and learn](/blog/2023/02/27/runiverse-discovering/) new
things. We are continuously growing this database by adding more R
projects, to guide you through everything the R ecosystem has to offer.

Currently r-universe is tracking and indexing of over 18.000 R packages.
These are a mix of packages found on popular networks like
[CRAN](https://cran.r-project.org/) or
[Bioconductor](https://www.bioconductor.org/), and packages that were
registered by users.

In previous posts we already explained [how to create your personal
CRAN-like repository](/blog/2021/06/22/setup-runiverse/) and publish
packages on r-universe yourself. This post explains the other part: how
the scraper automatically finds packages on CRAN and Bioconductor that
should be included in r-universe.

## Why we look for the upstream package source

For R packages to be trackable by r-universe, the source has to be
publicly accessible via Git^[1](#fn:1){.footnote-ref
role="doc-noteref"}^. Most packages in r-universe are found on GitHub,
but in fact any Git server is allowed.

We strongly prefer tracking projects from their official upstream Git
source, where the authors commit changes and where users report bugs.
The Git source provides a lot of useful information such as:

- The latest changes
- Who is the owner of the project
- If there is an active bug tracker
- Historical statistics on development activity and contributors
- Project metadata, such as keywords

R-universe automatically analyses all this information, uses it to rank
and classify packages, and presents the data via the
[r-universe.dev](https://r-universe.dev) web user interfaces and APIs.
For this reason we really want to know the official Git url and owner,
even when a copy of the package exists on CRAN or BioConductor.

## How we determine the upstream source

For all R packages on CRAN and BioConductor we perform the following
steps to try to find the upstream git source url:

1.  Inspect the `URL` and `BugReports` field in the DESCRIPTION file to
    look for a github/gitlab/bitbucket/r-forge url. If the package can
    be found here, this is the preferred method.
2.  If this fails, but the maintainer has a GitHub account, we look for
    the package under this Github account. If the package is found and
    the version is equal or greater than the version on CRAN or
    Bioconductor, this is treated as the official source.
3.  Finally if the maintainer has a Github account, but we could not
    find the package there either, we add a copy of the package from
    [metacran](https://github.com/cran) in the universe of the
    maintainer. As explained earlier, for this set of packages we do not
    have upstream metadata, so we can only index the package content and
    some maintainer information.

This list of package URLs is updated every night and published in
[crantogit](https://github.com/r-universe-org/cran-to-git/blob/master/crantogit.csv).
Today's statistics are:

- 10.805 CRAN/Bioc packages found at the Git url mentioned in the
  DESCRIPTION file (yay, you rule!)
- 1.983 packages found under the maintainer's personal Github account
- 4.613 packages ingested from the CRAN/Bioc mirror in the maintainer's
  universe

Currently we do not process CRAN/Bioc packages that have no public Git
source, and also the maintainer has no Github account, because we cannot
determine the owner (and hence r-universe subdomain).

This is roughly how it works, but there are some caveats. For example,
the scraper may not be able to find a package if it is stored in an
unusual subdirectory within a Git repository. Also, CRAN has an unusual
practice of unpredictably archiving and unarchiving packages. Therefore,
packages that get archived on CRAN and are also not part of any other
registry, still remain on r-universe for 2 months.

## Tips for package authors

If you maintain an R package, regardless of where you publish it, I
highly recommend these two things to let us (and others) identify the
official source and maintainer of the project:

- Put the Github/Gitlab/R-forge/Bitbucket home of your project in the
  `URL` and/or `BugReports` fields in the package DESCRIPTION file when
  you publish on CRAN/BioConductor^[2](#fn:2){.footnote-ref
  role="doc-noteref"}^. This makes it clear where to report bugs, and
  also prevents confusion about the official source if someone forks
  your package, or creates a package with the same name.
- If you have a GitHub account (even if you never use it!), do register
  your maintainer email address(es) in your [Github account
  settings](https://github.com/settings/emails) (see also [this
  faq](https://github.com/r-universe-org/help#how-to-link-a-maintainer-email-addresses-to-a-username-on-r-universe)).
  This way the maintainer can be linked to your github account/picture
  and systems like r-universe and metacran can correctly identify
  ownership and contributions.

Finally I want to emphasize again that packages do not need to be on
CRAN or Bioconductor to be included in r-universe. It is super easy to
[setup your own universe](/blog/2021/06/22/setup-runiverse/) and get the
same benefits!

::: {.footnotes role="doc-endnotes"}

------------------------------------------------------------------------

1.  ::: {#fn:1}
    One notable exception is [r-forge](https://r-forge.r-project.org/)
    which uses SVN, but has a live Git mirror on
    [github.com/r-forge](https://github.com/r-forge) [↩︎](#fnref:1){.footnote-backref
    role="doc-backlink"}
    :::

2.  ::: {#fn:2}
    You can do it manually or by running
    [`usethis::use_github_links()`](https://usethis.r-lib.org/reference/use_github_links.html). [↩︎](#fnref:2){.footnote-backref
    role="doc-backlink"}
    :::
:::


How packages appear in r-universe  Last month we explained how r-universe makes it easy to search and browse through the countless R packages, articles, and datasets to let you discover and learn new things. We are continuously growing this database by adding more R projects, to guide you through everything the R ecosystem has to offer. Currently r-universe is tracking and indexing of over 18.000 R packages.

How r-universe searches for packages on CRAN / Bioconductor

## Running your own package registry

Tiny update for [r-universe](https://r-universe.dev) users whos run
their own package registry: we have changed the preferred git repo name
where you host your `packages.json` registry file.

Previously the git repository was simply called `universe`, i.e. the
registry for
[tiledb-inc.r-universe.dev](https://tiledb-inc.r-universe.dev) was in
the GitHub repository `tiledb-inc/universe`. We now recommend to instead
use the full lowercase name of your r-universe domain as the repo name,
i.e
[`tiledb-inc/tiledb-inc.r-universe.dev`](https://github.com/tiledb-inc/tiledb-inc.r-universe.dev).

For now the old name `universe` will keep working as well, but this will
be deprecated in 2024 so we recommend renaming it now.

## Why the change

Why the change? First of all, the old name `universe` was too generic,
which made its purpose sometimes unclear for larger GitHub
organizations.

But there was also a potential bug: if you would fork somebody
`universe` repo in order to send them a PR, you would automatically have
a git repo named `universe` under your own GitHub account, and
unintentionally start a CRAN-like repo under your own name, but with
their packages.

Hence the obvious solution was to use the same format as github-pages,
where the name of the repo is simply the full domain name at which your
universe is hosted.


Running your own package registry  Tiny update for r-universe users whos run their own package registry: we have changed the preferred git repo name where you host your packages.json registry file. Previously the git repository was simply called universe, i.e. the registry for tiledb-inc.r-universe.dev was in the GitHub repository tiledb-inc/universe.

New preferred repo name for r-universe registries

## Pretty permanent URLs!

We have improved the URL schema for the
[r-universe.dev](https://r-universe.dev) front-end to make it even
easier to find and share a link to an R package, article, API, or other
resource.

The old URLs will now automatically be redirected to the new locations,
so this should be a non-breaking change. But of course we recommend
updating your references to link to the new permanent URLs.

### Package homepages

The new URLs have the same form as github pages, where the homepage for
a package can be found under the root directory of your universe url,
i.e. `https://{owner}.r-universe.dev/{pkg}`. For example the landing
packages for some packages look like this:

- [https://jeroen.r-universe.dev**/jsonlite**](https://jeroen.r-universe.dev/jsonlite)
- [https://ropensci.r-universe.dev**/targets**](https://ropensci.r-universe.dev/targets)
- [https://tidyverse.r-universe.dev**/ggplot2**](https://tidyverse.r-universe.dev/ggplot2)

And so on. The URLs above return package information in html format.
Simply append `/json` to retrieve the same information (and more) in api
format:

- [https://jeroen.r-universe.dev**/jsonlite/json**](https://jeroen.r-universe.dev/jsonlite/json)
- [https://ropensci.r-universe.dev**/targets/json**](https://ropensci.r-universe.dev/targets/json)
- [https://tidyverse.r-universe.dev**/ggplot2/json**](https://tidyverse.r-universe.dev/ggplot2/json)

A few exemptions are the following root paths, which are reserved names
for the r-universe front-end sections and cranlike-server, and hence do
not refer to a package:

- [https://jeroen.r-universe.dev**/builds**](https://jeroen.r-universe.dev/builds)
- [https://jeroen.r-universe.dev**/contributors**](https://jeroen.r-universe.dev/contributors)
- [https://jeroen.r-universe.dev**/packages**](https://jeroen.r-universe.dev/packages)
- [https://jeroen.r-universe.dev**/articles**](https://jeroen.r-universe.dev/articles)
- [https://jeroen.r-universe.dev**/badges**](https://jeroen.r-universe.dev/badges)
- [https://jeroen.r-universe.dev**/src**](https://jeroen.r-universe.dev/src)
  (cranlike-server)
- [https://jeroen.r-universe.dev**/bin**](https://jeroen.r-universe.dev/bin)
  (cranlike-server)
- [https://jeroen.r-universe.dev**/stats**](https://jeroen.r-universe.dev/stats)
  (stream apis)

### Resources within a package

Resources that are part of a package, such as the pdf/html manual, news
file, and preferred citation, can now be found under that package URL.
Some examples:

- [https://jeroen.r-universe.dev**/jsonlite/DESCRIPTION**](https://jeroen.r-universe.dev/jsonlite/DESCRIPTION)
- [https://jeroen.r-universe.dev**/jsonlite/NEWS**](https://jeroen.r-universe.dev/jsonlite/NEWS)
- [https://jeroen.r-universe.dev**/jsonlite/NEWS.txt**](https://jeroen.r-universe.dev/jsonlite/NEWS.txt)
- [https://jeroen.r-universe.dev**/jsonlite/jsonlite.pdf**](https://jeroen.r-universe.dev/jsonlite/jsonlite.pdf)
- [https://jeroen.r-universe.dev**/jsonlite/citation**](https://jeroen.r-universe.dev/jsonlite/citation)
  (if available)
- [https://jeroen.r-universe.dev**/jsonlite/citation.cff**](https://jeroen.r-universe.dev/jsonlite/citation.cff)
- [https://jeroen.r-universe.dev**/jsonlite/citation.txt**](https://jeroen.r-universe.dev/jsonlite/citation.txt)
- [https://jeroen.r-universe.dev**/jsonlite/citation.json**](https://jeroen.r-universe.dev/jsonlite/citation.json)
- [https://jeroen.r-universe.dev**/jsonlite/doc/**](https://jeroen.r-universe.dev/jsonlite/doc/)
  (html manual and vignettes)
- [https://jeroen.r-universe.dev**/jsonlite/doc/manual.html**](https://jeroen.r-universe.dev/jsonlite/doc/manual.html)
- [https://jeroen.r-universe.dev**/jsonlite/doc/json-mapping.pdf**](https://jeroen.r-universe.dev/jsonlite/doc/json-mapping.pdf)

The URLs for these resources are, where possible, consistent with the
CRAN website and/or the R local dynamic help server.

### Link to sections of a package homepage

One benefit of the new schema is that you can link directly to sections
within a package homepage. You can also find these links by clicking on
the header of the section on the page.

- [https://jeroen.r-universe.dev**/jsonlite#readme**](https://jeroen.r-universe.dev/jsonlite#readme)
- [https://jeroen.r-universe.dev**/jsonlite#reference**](https://jeroen.r-universe.dev/jsonlite#reference)
- [https://jeroen.r-universe.dev**/jsonlite#citation**](https://jeroen.r-universe.dev/jsonlite#citation)
- [https://jeroen.r-universe.dev**/jsonlite#development**](https://jeroen.r-universe.dev/jsonlite#development)
- [https://jeroen.r-universe.dev**/jsonlite#users**](https://jeroen.r-universe.dev/jsonlite#users)

### Feeds and sitemaps

Finally there are some meta APIs that are mainly for non-human
consumption:

- [https://jeroen.r-universe.dev**/feed.xml**](https://jeroen.r-universe.dev/feed.xml)
  (RSS feed)
- [https://jeroen.r-universe.dev**/sitemap_index.xml**](https://jeroen.r-universe.dev/sitemap_index.xml)
  (Universe sitemap index)
- [https://jeroen.r-universe.dev**/jsonlite/sitemap.xml**](https://jeroen.r-universe.dev/jsonlite/sitemap.xml)
  (package sitemap)

That's it for now. As said, in order to not break any existing links, we
have created automatic redirects from the old URLs to the new location
where possible. If you run into any problems, please open an issue in
[r-universe-org/help](https://github.com/r-universe-org/help) or send a
message on the `#r-universe` channel in the ropensci slack.


ul a:hover{text-decoration: underline;}Pretty permanent URLs! We have improved the URL schema for the r-universe.dev front-end to make it even easier to find and share a link to an R package, article, API, or other resource. The old URLs will now automatically be redirected to the new locations, so this should be a non-breaking change.

Improved permanent URL schema for r-universe.dev

University of Glasgow

Roderic Page

![Mattern 1 dashboard
1020x703](https://lh3.googleusercontent.com/-4EHuZyoyyKQ/VyCahO4mjLI/AAAAAAAAEME/WpkPNCUMpYk/mattern-1-dashboard-1020x703.jpg?imgmax=1600 "mattern-1-dashboard-1020x703.jpg"){style="display:block; margin-left:auto; margin-right:auto;"
border="0" width="460" height="317"} Despite the well deserved
scepticism about dashboards voiced by Shannon Mattern
[@shannonmattern](https://twitter.com/shannonmattern) (see [Mission
Control: A History of the Urban
Dashboard](https://placesjournal.org/article/mission-control-a-history-of-the-urban-dashboard/),
I discovered this by reading [Ignore the Bat Caves and Marketplaces:
lets talk about
Zoning](https://blog.ldodds.com/2016/02/21/ignore-the-bat-caves-and-marketplaces-lets-talk-about-zoning/)
by Leigh Dodds [@ldodds](https://twitter.com/ldodds)) I\'m intrigued by
the idea a dashboard for biodiversity. We could have several different
kinds of information, displayed in a single place.

### Immediate information

There are sites such as [Global Forest Watch
Fires](http://fires.globalforestwatch.org/home/) that track events that
affect biodiversity and which are haoppoening right now. Some of this
data can be harvested (e.g., from the NASA [Fire Information for
Resource Management System](https://firms.modaps.eosdis.nasa.gov)) to
show real-time forest fires. Below is an image for the last 24 hours:

::: {style="text-align:center;background-color:rgb(242,242,242);"}
![](https://firms.modaps.eosdis.nasa.gov/wms/c6/?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=fires24&width=1024&height=512&BBOX=-180,-90,180,90&&SRS=EPSG:4326){width="460"}
:::

We could also have Twitter feeds of these sorts of events

### Historical trends

We could have longer-term trends, such as [changes in forest
cover](http://www.globalforestwatch.org), or changes in abundance of
species over time.

### Trends in information

We could have feeds that show us how our knowledge is changing. For
example, we could have a map of data from the newest datasets uploaded
to GBIF, the lastest DNA barcodes, etc.

As an example, [@wikiredlist](https://twitter.com/wikiredlist) tweets
overtime an article about a species from the IUCN Red List is edited on
the English language Wikipedia.

[Tweets by
@wikiredlist](https://twitter.com/wikiredlist){.twitter-timeline
widget-id="725273894475698177"}

Imagine several such streams, both as lists and as maps. As another
example, a while ago I [created a visualisation of new species
discoveries](http://iphylo.blogspot.co.uk/2009/05/e-biosphere-challenge-visualising.html):

::::::: {style="text-align:center;"}
:::::: {.iframe}
::: {#player}
:::

:::: {.player-unavailable}
# Ein Fehler ist aufgetreten. {#ein-fehler-ist-aufgetreten. .message}

::: {.submessage}
[Sieh dir dieses Video auf www.youtube.com
an](https://www.youtube.com/watch?v=7aACcXyHCsM){target="_blank"} oder
aktiviere JavaScript, falls es in deinem Browser deaktiviert sein
sollte.
:::
::::
::::::
:::::::

### Summary

I\'m aware of the irony of drawing inspiration from a [critique of
dashboards](https://placesjournal.org/article/mission-control-a-history-of-the-urban-dashboard/),
but I still think there is value in having an overview of global
biodiversity. But we shouldn\'t loose site of the fact that such views
will be biassed and constrained, and in many cases it will be much easy
to visualise what is going on (or, at least, what our chosen sources
reveal) than to effect change on those trends that we find most
alarming.


Despite the well deserved scepticism about dashboards voiced by Shannon Mattern @shannonmattern (see Mission Control: A History of the Urban Dashboard, I discovered this by reading Ignore the Bat Caves and Marketplaces: lets talk about Zoning by Leigh Dodds @ldodds) I'm intrigued by the idea a dashboard for biodiversity. We could have several different kinds of information, displayed in a single place.

Rogue Scholar Posts

Downloading snapshots and creating stable R packages repositories using r-universe

How r-universe searches for packages on CRAN / Bioconductor

New preferred repo name for r-universe registries

Improved permanent URL schema for r-universe.dev

Possible project: Biodiversity dashboard