rOpenSci

Stefanie Butland

## rOpenSci HQ

Join our Community Manager [Stefanie Butland](/author/stefanie-butland/)
in a [CarpentryCon @ Home fireside chat on Growing Inclusive,
Computational Communities and
Leaders](https://2020.carpentrycon.org/schedule/#session-62). Live
90-min discussions Wed Aug 26 21h00 (2pm PDT) and Thu Aug 27 15h00 UTC
(8am PDT). Other panelists over the two sessions are Abigail Cabunoc
Mayes (Mozilla Foundation), Kate Hertweck (Fred Hutch Cancer Research
Center), Toby Hodges (The Carpentries), Angela Li (U Chicago), Marlene
Mhangami (Python Software Foundation), Naomi Penfold (eLife), and Serah
Rono (The Carpentries).
[Submit](https://docs.google.com/forms/d/e/1FAIpQLSc5bHp5nYG8XV1-WNQqZ_sm6h0T4G8Kp6M9biqbyRW1GrXNoA/viewform)
your questions for the panel before Aug 24.

## Software

2 new peer-reviewed packages from the community are on CRAN.

- **[chirps](https://docs.ropensci.org/chirps/)** - API Client for
  Climate Hazards Group InfraRed Precipitation with Station Data.
  Author: [Kauê de Sousa](https://github.com/kauedesousa)

- **[dittodb](https://docs.ropensci.org/dittodb/)** - Makes testing
  against databases easy. Author: [Jonathan
  Keane](/author/jonathan-keane/)

## On the Blog

### From the community

- [Developing dittodb](/blog/2020/08/18/dev-dittodb/) by [Jonathan
  Keane](/author/jonathan-keane/)

- [OSF: A Project Management Service Built for
  Research](/blog/2020/08/04/osf/) by [Aaron
  Wolen](/author/aaron-wolen/)

- [Maintaining an R Package - Community Call
  Summary](/blog/2020/08/18/dev-dittodb/) by [Janani
  Ravi](/author/janani-ravi/) and [Steffi
  LaZerte](/author/steffi-lazerte/)

- [Community Captioning of rOpenSci Community
  Calls](/blog/2020/07/15/subtitles/) by [Matthias
  Grenié](/author/matthias-greni%C3%A9/) and [Hugo
  Gruson](/author/hugo-gruson/)

### From the rOpenSci team

- [CRAN Checks API News: Documentation, Notifications, and
  More](/technotes/2020/07/16/cran-checks-docs-notifications/) Tech note
  by [Scott Chamberlain](/author/scott-chamberlain/) and [Maëlle
  Salmon](/author/ma%C3%ABlle-salmon/)

- [A better way to manage your Github personal access
  tokens](/technotes/2020/07/07/github-pat/) Tech note by [Jeroen
  Ooms](/author/jeroen-ooms/)

- [2 Months in 2 Minutes - rOpenSci News, June
  2020](/blog/2020/06/18/news-jun2020/) by [Stefanie
  Butland](/author/stefanie-butland/)

## Use Cases

- 124 published works cited or used rOpenSci software (listed in
  individual newsletters)

- 11 use cases for our packages or resources were posted in our
  [discussion forum](https://discuss.ropensci.org/c/usecases) Look for
  **av**, **DataPackageR**, **magick**, **pdftools**, **qualtRics**,
  **rtweet**, **tabulizer**, **webmockr**, **vcr**

*Have you used an rOpenSci package? [Share your use
case](https://discuss.ropensci.org/c/usecases) and we'll tweet about
it.*

## Keep up with rOpenSci

We create a newsletter every two weeks. You can subscribe via rss feed
in [XML](https://news.ropensci.org/feed.xml) or
[JSON](https://news.ropensci.org/feed.json) or via our one-way [mailing
list](/#subscribe).

Follow [@rOpenSci](https://twitter.com/ropensci) on Twitter.

Find out how you can
[contribute](https://devguide.ropensci.org/contributingguide.html) to
rOpenSci as a user or developer.


rOpenSci HQ   Join our Community Manager Stefanie Butland in a CarpentryCon @ Home fireside chat on Growing Inclusive, Computational Communities and Leaders.

2 Months in 2 Minutes - rOpenSci News, August 2020

Jeroen Ooms

The latest version of the rOpenSci [av
package](https://docs.ropensci.org/av) includes some useful new tools
for working with audio data. We have added functions for reading,
cutting, converting, transforming, and plotting audio data in any
popular audio / video format (mp3, mkv, aac, etc).

The functionality can either be used by itself, or to prepare audio data
for further analysis in R using other packages. We hope this clears an
important hurdle to use R for research on speech, music, and [whale
mating calls](https://youtu.be/F8Zt3mYlOqU).

## Audio analysis in R

CRAN has serveral cool packages for audio analysis, most notably the
[tuneR](https://CRAN.R-project.org/package=tuneR) package, maintained by
Uwe Ligges. However, getting your audio data into R can be tricky,
because real-world audio come in all sorts of formats and codecs, with
varying sampling rates, channel layouts, and so on.

I got stuck on this myself when trying to use sound from youtube videos,
or extract a short fragment from a long recording. The existing packages
mostly assume small wav files, but it is unclear where to get started to
analyze the complete oeuvre of Taylor Swift or a 3 hour recording of
bird sounds.

The av package builds on the
[FFmpeg](http://ffmpeg.org/documentation.html) libav libraries, which
provide a extensive, high performance implementations for streamable
reading, editing, and writing of media in any format. The av package is
already used in R for working with video, for example to create
[animated
graphics](https://ropensci.org/technotes/2018/10/06/av-release/) or
[sampling pictures from a camera
stream](https://ropensci.org/technotes/2019/09/27/ropensci-docs/) for
image analysis. By adding audio functionality, we hope to make it more
feasible to analyze real-world sound data directly in R.

## Converting between audio formats

The simplest use of `av` is converting an audio file into another
format. The
[av_audio_convert](https://docs.ropensci.org/av/reference/encoding.html)
function will convert any sound input (even from a video file) into
another output format, optionally adjusting the number of channels,
sampling rate, and start/end time.

Let's run some examples. First make sure you install the latest version
of the package from CRAN:

::: {.highlight}
``` {.chroma tabindex="0"}
install.packages("av")library(av)
```
:::

Suppose we want to read a certain piece of audio from a youtube
recording. We use the `youtube-dl` utility to download a full video from
youtube:

::: {.highlight}
``` {.chroma tabindex="0"}
# You need youtube-dl: https://ytdl-org.github.io/youtube-dl/download.html# MacOS: brew install youtube-dlsystem("youtube-dl https://youtu.be/F8Zt3mYlOqU -o whale.mp4")# Show some infoav_media_info('whale.mp4')#> $duration#> [1] 138.507#> #> $video#>   width height codec frames framerate  format#> 1  1280    720  h264   3460        25 yuv420p#> #> $audio#>   channels sample_rate codec frames bitrate layout#> 1        2       44100   aac   5965  125588 stereo
```
:::

This gives us a 75mb HD video with an AAC audio stream, which is not
something other packages can read. Suppose that for your analysis you
only need the first 10 seconds of sound, in mono `wav` format. Use
[av_audio_convert](https://docs.ropensci.org/av/reference/encoding.html)
to convert the input video into a shorter wav file:

::: {.highlight}
``` {.chroma tabindex="0"}
av_audio_convert('whale.mp4', 'whale10.wav', channels = 1, total_time = 10)#> Output #0, wav, to 'short.wav':#>   Metadata:#>     ISFT            : Lavf58.29.100#>     Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, mono, s16, 768 kb/s#>  Adding audio frame 503 at timestamp 10.01sec - audio stream completed!
```
:::

Now we have a nice small wav file that we can read with any of the
standard R packages. Note that this entire conversion above took 0.02
seconds and uses very little memory because it streams directly from the
input to the output file.

## Reading raw audio samples

For this example we use an MP3 demo that is included with the package.
The [av_media_info](https://docs.ropensci.org/av/reference/info.html)
function shows some information about this file (without reading it in
memory).

::: {.highlight}
``` {.chroma tabindex="0"}
# Our example datawonderland <- system.file('samples/Synapsis-Wonderland.mp3', package='av')av_media_info(wonderland)#> $duration#> [1] 30.04082#> #> $video#> NULL#> #> $audio#>   channels sample_rate    codec frames bitrate layout#> 1        2       44100 mp3float     NA  192000 stereo
```
:::

As we can see, this file has one audio stream and no video streams,
which is what we would expect. The function
[read_audio_bin](https://docs.ropensci.org/av/reference/read_audio.html)
reads audio files (in any format) into binary PCM samples. By setting
`end_time = 2` we only read the first two seconds (approximately),
keeping the data relatively small.

::: {.highlight}
``` {.chroma tabindex="0"}
pcm_data <- read_audio_bin(wonderland, channels = 1, end_time = 2.0)plot(pcm_data, type = 'l')
```
:::

![pcmplot](Ve6HraL.png)

The docs page for
[read_audio_bin](https://docs.ropensci.org/av/reference/read_audio.html)
shows parameters for reading audio, in particular to reduce the number
of channels (i.e. convert to mono) or downsample to make data a bit more
more managable.

## Frequency data and spectrograms

Raw data is not very informative, most analyses require that the signal
is transformed into frequency data. The
[read_audio_fft](https://docs.ropensci.org/av/reference/read_audio.html)
function converts audio directly into frequency data using FFmpeg's
built-in FFT. Because the FFT is done on-the-fly when reading the audio,
it requires relatively little memory. It returns a matrix with the
time-frequency data, from which we can plot the spectrogram.

::: {.highlight}
``` {.chroma tabindex="0"}
# Read 5 sec of data and directly transform to frequencyfft_data <- read_audio_fft(wonderland, end_time = 5.0)dim(fft_data)#> [1] 512 860# Plot the spectrogramplot(fft_data)
```
:::

![fftplot](https://docs.ropensci.org/av/reference/read_audio-1.png)

The docs page of the av package has an entire article dedicated to
[spectograms](https://docs.ropensci.org/av/articles/articles/spectrograms.html),
comparing the various R packages. This vignette also shows the
[av_spectrogram_video](https://docs.ropensci.org/av/reference/capturing.html)
function which generates a video that overlays the audio on an animated
spectrogram with moving status bar:

::: {.highlight}
``` {.chroma tabindex="0"}
# Create new audio file with first 5 secav_audio_convert(wonderland, 'short.mp3', total_time = 5)av_spectrogram_video('short.mp3', output = 'spectrogram.mp4', width = 1280, height = 720, res = 144)
```
:::

Your browser does not support the video tag.


The latest version of the rOpenSci av package includes some useful new tools for working with audio data. We have added functions for reading, cutting, converting, transforming, and plotting audio data in any popular audio / video format (mp3, mkv, aac, etc).  The functionality can either be used by itself, or to prepare audio data for further analysis in R using other packages.

Working with audio in R using av

At rOpenSci we are developing on a suite of packages that expose
powerful graphics and imaging libraries in R. Our latest addition is
[av](https://github.com/ropensci/av#readme) -- a new package for working
with audio/video based on the [FFmpeg](https://www.ffmpeg.org/) AV
libraries. This ambitious new project will become the video counterpart
of the
[magick](https://cloud.r-project.org/web/packages/magick/vignettes/intro.html)
package which we use for working with images.

::: {.highlight}
``` {.chroma tabindex="0"}
install.packages("av")av::av_demo()
```
:::

The package can be installed directly from CRAN and includes a test
function `av_demo()` which generates a demo video from random
histograms.

## Why AV in R?

One popular application is animating graphics by combining a sequence of
graphics into a video. The
[animation](https://cran.r-project.org/web/packages/animation/index.html)
and [gganimate](https://github.com/thomasp85/gganimate) packages have
many great examples. However up till now these packages would have to
shell out to external software (such as the `ffmpeg` command line
program) to generate the video. This process is inefficient and error
prone and requires that the correct version of the external software is
installed on the user/server machines, which is often not the case.

The `av` package takes away this technical burden. It uses the same
libraries as FFmpeg, however because we interface directly to the C API,
there is no need to shell out or install utilities. Everything we need
is linked into the R package, which means that if the package is
installed, it always works.

FFmpeg provides a full-featured video editing library, and now that the
core package is in place, we can take things a step further. For example
you can already enhance an animation with an audio track (to narrate
what is going on or show off your karaoke skills) or apply one of the
100+ built-in [video
filters](https://ffmpeg.org/ffmpeg-filters.html#Video-Filters). In
future versions we also want to add things like screen capturing and
reading raw video frames and audio samples for analysis in R.

## Create video from images

The `av_encode_video()` function converts a set of images (png, jpeg,
etc) into a video file with custom container format, codec, fps, and
filters. The video format is determined from the file extension (mp4,
mkv, flv, gif). Av supports all popular codecs and muxers (codecs
compress the raw audio/video and a muxer is the container format which
interleaves one or more audio and video streams into a file).

::: {.highlight}
``` {.chroma tabindex="0"}
# Create some PNG imagespng("input%03d.png", width = 1280, height = 720, res = 108)for(i in 1:10){  print(ggplot2::qplot(rnorm(100)))}dev.off()png_files <- sprintf("input%03d.png", 1:10)av::av_encode_video(png_files, 'output.mp4', framerate = 1)utils::browseURL('output.mp4')
```
:::

Your browser does not support the video tag.

## Create video from graphics

The example above illustrates how to generate encode a set of png images
into a HD video. For generating a video from the R graphics, av includes
`av_capture_graphics()` -- a convenient wrapper which automatically
opens and closes the graphics device and then encodes the video:

::: {.highlight}
``` {.chroma tabindex="0"}
library(gapminder)library(ggplot2)makeplot <- function(){  datalist <- split(gapminder, gapminder$year)  lapply(datalist, function(data){    p <- ggplot(data, aes(gdpPercap, lifeExp, size = pop, color = continent)) +      scale_size("population", limits = range(gapminder$pop)) + geom_point() + ylim(20, 90) +      scale_x_log10(limits = range(gapminder$gdpPercap)) + ggtitle(data$year) + theme_classic()    print(p)  })}# Play 1 plot per secvideo_file <- file.path(tempdir(), 'output.mp4')av::av_capture_graphics(makeplot(), video_file, 1280, 720, res = 144)utils::browseURL(video_file)
```
:::

Your browser does not support the video tag.

## Using av with gganimate

The latest version of the super cool gganimate package already has
built-in support for rendering video with av by using the
`av_renderer()` output function. It is not yet on CRAN so you need to
install from GitHub:

::: {.highlight}
``` {.chroma tabindex="0"}
# Install gganimatedevtools::install_github("thomasp85/gganimate", "thomasp85/transformr")
```
:::

Try this example animation:

::: {.highlight}
``` {.chroma tabindex="0"}
library(gganimate)# Create the gganimate plotp <- ggplot(airquality, aes(Day, Temp)) +   geom_line(size = 2, colour = 'steelblue') +   transition_states(Month, 4, 1) +   shadow_mark(size = 1, colour = 'grey')# Render and show the videodf <- animate(p, renderer = av_renderer('animation.mp4'),  width = 1280, height = 720, res = 104, fps = 25)utils::browseURL('animation.mp4')
```
:::

Your browser does not support the video tag.

Check the [gganimate repo](https://github.com/thomasp85/gganimate) for
many cool examples.

## Filters

AV also allows for adding a custom [video filter
chain](https://ffmpeg.org/ffmpeg-filters.html#Video-Filters). For
example this will use the same animation as above, then negate the
colors, and apply an orange fade-in effect to the first 15 frames.

::: {.highlight}
``` {.chroma tabindex="0"}
# Continue on the example abovemyrenderer <- av_renderer('animation.mp4',  vfilter = 'negate=1, fade=in:0:15:color=orange')df <- animate(p, renderer = myrenderer,   width = 1280, height = 720, res = 104, fps = 25)utils::browseURL('animation.mp4')
```
:::

Your browser does not support the video tag.

Note that filters can affect the number of frames and final framerate of
the video. Below the same example as we had earlier, however now we
added a `vfilter` which increases the framerate of the video from 1 to
10 by interpolating the intermediate frames, which results in a smoother
transition between the frames:

::: {.highlight}
``` {.chroma tabindex="0"}
library(gapminder)library(ggplot2)makeplot <- function(){  datalist <- split(gapminder, gapminder$year)  lapply(datalist, function(data){    p <- ggplot(data, aes(gdpPercap, lifeExp, size = pop, color = continent)) +      scale_size("population", limits = range(gapminder$pop)) + geom_point() + ylim(20, 90) +      scale_x_log10(limits = range(gapminder$gdpPercap)) + ggtitle(data$year) + theme_classic()    print(p)  })}# Play 1 input plot per sec, interpolate frames until 10 FPSav::av_capture_graphics(makeplot(), 'smooth.mp4', 1280, 720, res = 104,    vfilter = 'framerate=fps=10')utils::browseURL('smooth.mp4')
```
:::

Your browser does not support the video tag.


At rOpenSci we are developing on a suite of packages that expose powerful graphics and imaging libraries in R. Our latest addition is av – a new package for working with audio/video based on the FFmpeg AV libraries. This ambitious new project will become the video counterpart of the magick package which we use for working with images.

Messaggi di Rogue Scholar

2 Months in 2 Minutes - rOpenSci News, August 2020

Working with audio in R using av

The av Package: Production Quality Video in R