Jeroen Ooms

Thomas Lin Pedersen

Release 1.4 of the [magick
package](https://cran.r-project.org/web/packages/magick/vignettes/intro.html)
introducesa new feature called [image
convolution](https://en.wikipedia.org/wiki/Kernel_(image_processing)#Convolution)
thatwas requested by Thomas L. Pedersen. In this post we explain what
this is all about.

## Kernel Matrix

The new `image_convolve()` function applies a
[kernel](https://en.wikipedia.org/wiki/Kernel_(image_processing)) over
the image. Kernel convolution means that each pixel value is
recalculated using the *weighted neighborhood sum* defined in the kernel
matrix. For example lets look at this simple kernel:

::: {.highlight}
``` {.chroma tabindex="0"}
library(magick)kern <- matrix(0, ncol = 3, nrow = 3)kern[1, 2] <- 0.25kern[2, c(1, 3)] <- 0.25kern[3, 2] <- 0.25kern##      [,1] [,2] [,3]## [1,] 0.00 0.25 0.00## [2,] 0.25 0.00 0.25## [3,] 0.00 0.25 0.00
```
:::

This kernel changes each pixel to the mean of its horizontal and
vertical neighboring pixels, which results in a slight blurring effect
in the right-hand image below:

::: {.highlight}
``` {.chroma tabindex="0"}
img <- image_read('logo:')img_blurred <- image_convolve(img, kern)image_append(c(img, img_blurred))
```
:::

![image_appended](Y6xByUL.gif)

## Standard Kernels

Many operations in `magick` such as blurring, sharpening, and edge
detection areactually special cases of image convolution. The benefit of
explicitly using`image_convolve()` is more control. For example, we can
blur an image and then blendit together with the original image in one
step by mixing a blurring kernel with theunit kernel:

::: {.highlight}
``` {.chroma tabindex="0"}
img %>% image_convolve('Gaussian:0x5', scaling = '60,40%')
```
:::

![mixed](6Vf6c2hl.gif)

The above requires a bit of explanation. ImageMagick defines several
common[standard kernels](https://www.imagemagick.org/Usage/convolve/)
such as thegaussian kernel. Most of the standard kernels take one or
more parameters,e.g. the example above used a gaussian kernel with 0
*radius* and 5 *sigma*.

In addition, `scaling` argument defines the magnitude of the kernel, and
possiblyhow much of the original picture should be mixed in. Here we mix
60% of theblurring with 40% of the original picture in order to get a
diffused lightning effect.

## Edge Detection

Another area where kernels are of use is in edge detection. A simple
example ofa direction-aware edge detection kernel is the
[*Sobel*](https://en.wikipedia.org/wiki/Sobel_operator) kernel.As can be
seen below, vertical edges are detected while horizontals are not.

::: {.highlight}
``` {.chroma tabindex="0"}
img %>% image_convolve('Sobel') %>% image_negate()
```
:::

![edges](i8ndfCu.gif)

Something less apparent is that the result of the edge detection is
truncated.Edge detection kernels can result in negative color values
which get truncated to zero.To combat this it is possible to add a
`bias` to the result. Often you'll end up withscaling the kernel to 50%
and adding 50% bias to move the midpoint of the result to 50%grey:

::: {.highlight}
``` {.chroma tabindex="0"}
img %>% image_convolve('Sobel', scaling = '50%', bias = '50%')
```
:::

![50pct](llUawrg.gif)

## Sharpening

ImageMagick has many more edge detection kernels, some of which are
insensitive tothe direction of the edge. To emulate a classic high-pass
filter from photoshop use[difference of
gaussians](https://en.wikipedia.org/wiki/Difference_of_Gaussians)
kernel:

::: {.highlight}
``` {.chroma tabindex="0"}
img %>% image_convolve('DoG:0,0,2') %>% image_negate()
```
:::

![dog](o5kODpc.gif)

As with the blurring, the original image can be blended in with the
transformed one, effectively sharpening the image along edges.

::: {.highlight}
``` {.chroma tabindex="0"}
img %>% image_convolve('DoG:0,0,2', scaling = '100, 100%')
```
:::

![combination](MtcMSn7.gif)

The [ImageMagick
documentation](https://www.imagemagick.org/Usage/convolve/) has more
examples of convolve with various avaiable kernels.


Release 1.4 of the magick package introducesa new feature called image convolution thatwas requested by Thomas L. Pedersen. In this post we explain what this is all about.Kernel Matrix  The new image_convolve() function applies a kernel over the image. Kernel convolution means that each pixel value is recalculated using the
<em>
 weighted neighborhood sum
</em>
defined in the kernel matrix.

Image Convolution in R using Magick

Last week, version 1.0 of the
[magick](https://cran.r-project.org/web/packages/magick/index.html)
package appeared on CRAN: an ambitious effort to modernize and simplify
high quality image processing in R. This R package builds upon the
[Magick++ STL](https://www.imagemagick.org/Magick++/STL.html) which
exposes a powerful C++ API to the famous ImageMagick library.

![RStudio Screenshot](https://jeroen.github.io/images/magick.png)

The best place to start learning about magick is the
[vignette](https://cran.r-project.org/web/packages/magick/vignettes/intro.html)
which gives a brief overview of the overwhelming amount of functionality
in this package.

### Towards Release 1.0

Last year around this time rOpenSci
[announced](/blog/2016/08/23/z-magick-release) the first release of the
magick package: a new powerful toolkit for image reading, writing,
converting, editing, transformation, annotation, and animation in R.
Since the initial release there have been several updates with
additional functionality, and many useRs have started to discover the
power of this package to take visualization in R to the next level.

For example [Bob
Rudis](https://twitter.com/hrbrmstr/status/758304420224466944) uses
magick to visualize California drought data from the U.S. Drought
Monitor (click on the image to go find out more):

[![drought](https://jeroen.github.io/images/drought.gif)](https://rud.is/b/2016/07/27/u-s-drought-animations-with-the-witchs-brew-purrr-broom-magick/)

R-ladies [Lucy D'Agostino McGowan](https://www.lucymcgowan.com/) and
[Maëlle Salmon](https://masalmon.eu/) demonstrate how to make a
beautiful collage:

[![collage](https://livefreeordichotomize.com/images/we-r-ladies.jpeg)](https://livefreeordichotomize.com/2017/07/18/the-making-of-we-r-ladies/)

And [Daniel P.
Hadley](https://twitter.com/danielphadley/status/884845188979359744)
lets Vincent Vega explains Cars:

[![travolta](https://danielphadley.com/images/Cars_Travolta.gif)](https://danielphadley.com/ggplot-Logo/)

Now, 1 year later, the 1.0 release marks an important milestone: the
addition of a new native graphics device (which serves as a hybrid
between a magick image object and an R plot) bridges the gap between
graphics and image processing in R.

This blog post explains how the magick device allows you to seamlessly
combine graphing with image processing in R. You can either use it to
post-process your R graphics, or draw on imported images using the
native R plotting machinery. We hope that this unified interface will
make it easier to produce beautiful, reproducible images with R.

### Native Magick Graphics

The `image_graph()` function opens a new graphics device similar to e.g.
`png()` or `x11()`. It returns an image object to which the plot(s) will
be written. Each page in the plotting device will become a frame (layer)
in the image object.

::: {.highlight}
``` {.chroma tabindex="0"}
# Produce image using graphics devicefig <- image_graph(res = 96)ggplot2::qplot(mpg, wt, data = mtcars, colour = cyl)dev.off()
```
:::

The `fig` object now contains the image that we can easily post-process.
For example we can overlay another image:

::: {.highlight}
``` {.chroma tabindex="0"}
logo <- image_read("https://www.r-project.org/logo/Rlogo.png")out <- image_composite(fig, image_scale(logo, "x150"), offset = "+80+380")# Show previewimage_browse(out)# Write to fileimage_write(out, "myplot.png")
```
:::

![out](/assets/blog-images/2017-08-15-magick-10/out.png)

### Drawing Device

The `image_draw()` function opens a graphics device to draw on top of an
existing image using pixel coordinates.

::: {.highlight}
``` {.chroma tabindex="0"}
# Open a filelibrary(magick)frink <- image_read("https://jeroen.github.io/images/frink.png")drawing <- image_draw(frink)
```
:::

![frink](https://jeroen.github.io/images/frink.png)

We can now use R's native low-level graphics functions for drawing on
top of the image:

::: {.highlight}
``` {.chroma tabindex="0"}
rect(20, 20, 200, 100, border = "red", lty = "dashed", lwd = 5)abline(h = 300, col = 'blue', lwd = '10', lty = "dotted")text(10, 250, "Hoiven-Glaven", family = "courier", cex = 4, srt = 90)palette(rainbow(11, end = 0.9))symbols(rep(200, 11), seq(0, 400, 40), circles = runif(11, 5, 35),  bg = 1:11, inches = FALSE, add = TRUE)
```
:::

At any point you can inspect the current result:

``` {tabindex="0"}
image_browse(drawing)
```

![drawing](/assets/blog-images/2017-08-15-magick-10/drawing.png)

Once you are done you can close the device and save the result.

::: {.highlight}
``` {.chroma tabindex="0"}
dev.off()image_write(drawing, 'drawing.png')
```
:::

By default `image_draw()` sets all margins to 0 and uses graphics
coordinates to match image size in pixels (width x height) where (0,0)
is the top left corner. Note that this means the y axis increases from
top to bottom which is the opposite of typical graphics coordinates. You
can override all this by passing custom `xlim`, `ylim` or `mar` values
to `image_draw()`.

### Animated Graphics

The graphics device supports multiple frames which makes it easy to
create animated graphics. The example below shows how you would
implement the example from the very cool
[gganimate](https://github.com/dgrtwo/gganimate) package using the
magick.

::: {.highlight}
``` {.chroma tabindex="0"}
library(gapminder)library(ggplot2)library(magick)img <- image_graph(res = 96)datalist <- split(gapminder, gapminder$year)out <- lapply(datalist, function(data){  p <- ggplot(data, aes(gdpPercap, lifeExp, size = pop, color = continent)) +    scale_size("population", limits = range(gapminder$pop)) +    scale_x_log10(limits = range(gapminder$gdpPercap)) +    geom_point() + ylim(20, 90) +  ggtitle(data$year) + theme_classic()  print(p)})dev.off()animation <- image_animate(img, fps = 2)image_write(animation, "animation.gif")
```
:::

![animation](/assets/blog-images/2017-08-15-magick-10/animation.gif)

We hope that the magick package can provide a more robust back-end for
packages like gganimate to produce interactive graphics in R without
requiring the user to manually install external image editing software.

### Porting ImageMagick Commands to R

The magick 1.0 release now has the core image processing functionality
that you expect from an image processing package. But there is still a
lot of room for improvement to make magick *the* image processing
package in R.

A lot of R users and packages currently shell out to ImageMagick command
line tools for performing image manipulations. The goal is to support
all these operations in the magick package, so that the images can be
produced (and reproduced!) on any platform without requiring the user to
install additional software.

Note that ImageMagick library is over 26 years old and has accumulated
an enormous number of features in those years. Porting all of this to R
is quite a bit of work, for which feedback from users is important. If
there is an imagemagick operation that you like to do in R but you can't
figure out how, please [open an
issue](https://github.com/ropensci/magick/issues) on GitHub. If the
functionality is currently not supported yet, we will try to add it to
the next version.

### Image Analysis

Currently magick is focused on generating and editing images. There is
yet another entirely different set of features which we like to support
related to analyzing images. Image analysis can involve anything from
calculating color distributions to more sophisticated feature extraction
and vision tools. I am not very familiar with this field, so again we
could use suggestions from users and experts.

One feature that is already available is the `image_ocr()` function
which extracts text from the image using the rOpenSci
[tesseract](/blog/2016/11/16/tesseract) package. Another cool example of
using image analysis is the
[collage](https://github.com/ThinkRstat/collage) package which
calculates [color
histograms](https://github.com/ThinkRstat/collage#histograms) to select
appropriate tile images for creating a collage.

[![histogram](https://github.com/ThinkRstat/collage/raw/master/README-histograms-2.png)](https://github.com/ThinkRstat/collage#histograms)

As part of supporting supporting analysis tools we plan to extract the
bitmap (raster) classes into a separate package. This will enable
package authors to write R extensions to analyze and manipulate on the
raw image data, without necessarily depending on magick. Yet the user
can always rely on magick as a powerful toolkit to import/export images
and graphics into such low level bitmaps.

> The greatest achievement of my PhD so far: discovering the \'magick\'
> R package and adding gifs to my plots (thanks
> [@danielphadley](https://twitter.com/danielphadley?ref_src=twsrc%5Etfw)!)
> [\#rstats](https://twitter.com/hashtag/rstats?src=hash&ref_src=twsrc%5Etfw)
> [pic.twitter.com/gCEfKyWa9x](https://t.co/gCEfKyWa9x)
>
> --- George Bailey (@grbails) [July 13,
> 2017](https://twitter.com/grbails/status/885543687559811073?ref_src=twsrc%5Etfw)


Last week, version 1.0 of the magick package appeared on CRAN: an ambitious effort to modernize and simplify high quality image processing in R. This R package builds upon the Magick++ STL which exposes a powerful C++ API to the famous ImageMagick library.

Messages de Rogue Scholar

Image Convolution in R using Magick

Magick 1.0: 🎩 ✨🐇 Advanced Graphics and Image Processing in R