Maëlle Salmon

Scott Chamberlain

rOpenSci

Stefanie Butland

Last year we reported on [the joy of using commonmark and xml2 to
parseMarkdown content, like the source of this website built
withHugo](/technotes/2018/09/05/commonmark/), in particular [to
extractlinks](https://ropensci.org/technotes/2018/09/05/commonmark/#urls-parsing),at
the time merely to count them. How about we go a bit further and usethe
same approach to find links to be fixed? In this tech note we
shallreport our experience using R to find broken/suboptimal links and
fixthem.

## What is a bad URL?

We tackled a few URL issues:

- We had used [absolute links (using our domain name) instead
  ofrelativelinks](https://stackoverflow.com/questions/2005079/absolute-vs-relative-urls).`https://ropensci.org/blog/`
  should be `/blog/`.

- Some internal and external links were broken, but we did not knowwhich
  ones.

- A few links were short links (`bit.ly/blabla`) whereas it's best
  tostore the actual link because the short link could break too.

- Some links were http links, although the same https link might workand
  would be preferred over http for security.

- There were links to ropensci.github.io documentation websites thatcan
  be replaced with links to [our brand-new
  docsserver](/technotes/2019/06/07/ropensci-docs/).

## Please read if you try this at "home/"

There are three main ingredients to our website spring/fall cleaning:
Rtools, elbow grease and *version control*! Most changes happened in
abranch, and although one can't possibly look in detail at a diff of
morethan one hundred files, we tried to be as careful as possible.

## From absolute to relative links

To remove the absolute links, we resorted to using regular expressions.

::: {.highlight}
``` {.chroma tabindex="0"}
library("magrittr")# Identify the Markdown files to be examinedmds <- fs::dir_ls("content", recurse = TRUE, glob = "*.md")mds <- mds[!grepl("\\/tutorials\\/", mds)]# Function to fix each file if neededfix_ropensci <- function(filepath){  readLines(filepath) -> text    # We only edit files that had the issue  if (any(grepl("http(s)?\\:\\/\\/ropensci\\.org\\/", text))){    text %>%      stringr::str_replace_all("http(s)?\\:\\/\\/ropensci\\.org\\/", "/") %>%      writeLines(filepath)  }}purrr::walk(mds, fix_ropensci)
```
:::

Voilà!

## Broken URLs

Now, what about the links that do not link to anything? We started
byextracting all links together with the relevant file paths.

::: {.highlight}
``` {.chroma tabindex="0"}
library("magrittr")website_source <- "/home/maelle/Documents/ropensci/roweb2"mds <- fs::dir_ls(website_source, recurse = TRUE, glob = "*.md")mds <- mds[!grepl("\\/tutorials\\/", mds)]get_links <- function(filepath){  readLines(filepath) %>%    glue::glue_collapse(sep = "\n") %>%    commonmark::markdown_html(normalize = TRUE,                              extensions = TRUE) %>%    xml2::read_html() %>%    xml2::xml_find_all("//a") %>%    xml2::xml_attr("href") -> urls    tibble::tibble(filepath = filepath,                 url = urls)}all_urls <- purrr::map_df(mds, get_links)all_urls <- all_urls  %>%  dplyr::mutate(url = stringr::str_remove_all(url, "#.*"),                url = stringr::str_remove(url, "\\/$"))all_urls
```
:::

    ## # A tibble: 14,234 x 2##    filepath                                            url                      ##    <chr>                                               <chr>                    ##  1 /home/maelle/Documents/ropensci/roweb2/content/aut… https://adamhsparks.netl…##  2 /home/maelle/Documents/ropensci/roweb2/content/aut… https://aldocompagnoni.w…##  3 /home/maelle/Documents/ropensci/roweb2/content/aut… http://robitalec.ca      ##  4 /home/maelle/Documents/ropensci/roweb2/content/aut… https://alison.rbind.io  ##  5 /home/maelle/Documents/ropensci/roweb2/content/aut… https://dobb.ae          ##  6 /home/maelle/Documents/ropensci/roweb2/content/aut… https://thestudyofthehou…##  7 /home/maelle/Documents/ropensci/roweb2/content/aut… https://annakrystalli.me ##  8 /home/maelle/Documents/ropensci/roweb2/content/aut… https://paleantology.com…##  9 /home/maelle/Documents/ropensci/roweb2/content/aut… https://aurielfournier.g…## 10 /home/maelle/Documents/ropensci/roweb2/content/aut… https://faculty.washingt…## # … with 14,224 more rows

We chose a different method to find those within and outside of
ourwebsite.

### Broken internal URLs

When building a Hugo website, one gets a sitemap, which is basically
acollection of links to all the pages of the website. [If an
internallink is not in the sitemap, it does not
exist.](https://masalmon.eu/404)

We generated the sitemap from within the website folder to extractlinks.

::: {.highlight}
``` {.chroma tabindex="0"}
cwd <- getwd()setwd(website_source)p <- processx::process$new("hugo", args = "server", echo = TRUE)
```
:::

    ## Running hugo server

::: {.highlight}
``` {.chroma tabindex="0"}
Sys.sleep(120)localhost <- "http://localhost:1313"browseURL(localhost)paste0(localhost, "/sitemap.xml") %>%  xml2::read_xml() %>%  xml2::xml_ns_strip() %>%  xml2::xml_find_all("//loc") %>%  xml2::xml_text() %>%  stringr::str_remove_all(localhost) %>%  stringr::str_remove("\\/$") -> linksp$kill()
```
:::

    ## [1] TRUE

::: {.highlight}
``` {.chroma tabindex="0"}
setwd(cwd)head(links)
```
:::

    ## [1] "/authors/scott-chamberlain"         "/tags/api"                         ## [3] "/authors"                           "/tags/http"                        ## [5] "/technotes/2019/12/11/http-testing" "/tags/mocking"

So these are the existing internal links. We could also have
extractedthem using [the multi-request features
ofcurl](https://github.com/jeroen/curl/blob/master/examples/sitemap.R).

Let's now extract the internal links we used in the content.

::: {.highlight}
``` {.chroma tabindex="0"}
all_urls %>%  dplyr::filter(!grepl("^http", url)) ->   internal_urlshead(internal_urls)
```
:::

    ## # A tibble: 6 x 2##   filepath                                             url                      ##   <chr>                                                <chr>                    ## 1 /home/maelle/Documents/ropensci/roweb2/content/blog… /community               ## 2 /home/maelle/Documents/ropensci/roweb2/content/blog… /community               ## 3 /home/maelle/Documents/ropensci/roweb2/content/blog… /blog/2013/05/10/introdu…## 4 /home/maelle/Documents/ropensci/roweb2/content/blog… /about                   ## 5 /home/maelle/Documents/ropensci/roweb2/content/blog… /community               ## 6 /home/maelle/Documents/ropensci/roweb2/content/blog… /contact

So, what are the missing ones? We used the code below to identify
themand then we [manually fixed or
removedthem](https://github.com/ropensci/roweb2/pull/534/commits/7715dc022141888b4f064cc44fe50b29b031c755).

::: {.highlight}
``` {.chroma tabindex="0"}
 internal_urls %>%   dplyr::filter(!url %in% links)
```
:::

### Broken external URLs

To identify broken external URLs, we
ran[`crul::ok()`](https://docs.ropensci.org/crul/reference/ok.html) on
allof them and created a big spreadsheet of URLs to look at.

::: {.highlight}
``` {.chroma tabindex="0"}
external_urls <- dplyr::anti_join(all_urls, internal_urls,                                   by = c("filepath", "url"))unique_urls <- unique(external_urls[, "url"])ok <- memoise::memoise(  ratelimitr::limit_rate(crul::ok,                          ratelimitr::rate(1, 1)))get_ok <- function(url){  message(url)  ok(url)}unique_urls <- unique_urls %>%  dplyr::group_by(url) %>%  dplyr::summarise(ok = get_ok(url))external_urls <- dplyr::left_join(external_urls, unique_urls,                                  by = "url")external_urls <- dplyr::arrange(external_urls, url)parse_one_post <- function(path){  if (grepl("\\_index", path)){    return(NULL)  }  lines <- suppressWarnings(readLines(path, encoding = "UTF-8"))  yaml <- blogdown:::split_yaml_body(lines)$yaml  yaml <- glue::glue_collapse(yaml, sep = "\n")  yaml <- yaml::yaml.load(yaml)    meta <- tibble::tibble(date = anytime::anydate(yaml$date),                         author = yaml$authors,                         title = yaml$title,                          software_peer_review = "Software Peer Review" %in% yaml$tags,                         type = dplyr::if_else(grepl("\\/blog\\/", path),                                               "blog post", "tech note"),                         filepath = path)    meta}info <- purrr::map_df(mds[grepl("blog", mds)|grepl("technotes",mds)], parse_one_post)info <- dplyr::group_by(info, filepath) %>%  dplyr::summarize(date = date[1],                   author = toString(author),                   title = title[1],                   type = type[1])bad_urls <- dplyr::filter(external_urls, !ok)bad_urls <- dplyr::left_join(bad_urls, info, by = "filepath")readr::write_csv(bad_urls, "urls.csv")
```
:::

From that spreadsheet hundreds of links were examined **manually**!
Whenthere was a replacement link, we used it thanks to a code looping
overall links. For the about 50 links without replacement, we amended
theposts by hand to make sure to take context into account
(e.g. removingthe link vs. removing the whole sentence presenting it).

There were quite a few false positives i.e. actually valid URLs.
Thislead to some edits
in[`crul::ok()`](https://docs.ropensci.org/crul/reference/ok.html) and
thefollowing wisdom:

Sometimes you'll get an error for the HEAD request but not the
GETrequest.

::: {.highlight}
``` {.chroma tabindex="0"}
# use get verb instead of headcrul::ok("http://animalnexus.ca")
```
:::

    ## [1] FALSE

::: {.highlight}
``` {.chroma tabindex="0"}
crul::ok("http://animalnexus.ca", verb = "get")
```
:::

    ## [1] TRUE

Sometimes you'll need an user-agent whose name does not contain
"curl",which the default user-agent of crul contains (`crul:::make_ua()`
islibcurl/7.58.0 r-curl/4.3 crul/0.9.1.9991).

::: {.highlight}
``` {.chroma tabindex="0"}
# some urls will require a different useragent string# they probably regex the useragent stringcrul::ok("https://doi.org/10.1093/chemse/bjq042")
```
:::

    ## GnuTLS recv error (-54): Error in the pull function.## [1] FALSE

::: {.highlight}
``` {.chroma tabindex="0"}
crul::ok("https://doi.org/10.1093/chemse/bjq042", verb = "get", useragent = "foobar")
```
:::

    ## [1] TRUE

## From short to long links

We only identified short links using the bit.ly service. We found
thecorresponding link by running the function below. There were
actuallyonly 4 short links so that was quick.

::: {.highlight}
``` {.chroma tabindex="0"}
get_long <- function(url){  crul::HttpClient$new(url)$get()$url}get_long("http://bit.ly/2JfrzmE")
```
:::

    ## [1] "https://www.timeanddate.com/worldclock/fixedtime.html?msg=rOpenSci+Community+Call+on+Reproducible+Research+with+R&iso=20190730T09&p1=791&ah=1"

## http vs https

> HTTPS: HTTP + security
> [pic.twitter.com/pkk7ZVzjz3](https://t.co/pkk7ZVzjz3)
>
> --- 🔎Julia Evans🔍 (@b0rk) [August 9,
> 2019](https://twitter.com/b0rk/status/1159812119099060224?ref_src=twsrc%5Etfw)

We proceeded as previously when checking external links, except we
usedbetter settings for `crul::ok()`.

::: {.highlight}
``` {.chroma tabindex="0"}
http <- dplyr::filter(all_urls, grepl("http\\:", url))http <- dplyr::mutate(http, https = sub("http\\:", "https:", url))unique_urls <- unique(http[, "https"])ok <- memoise::memoise(  ratelimitr::limit_rate(crul::ok,                          ratelimitr::rate(1, 1)))get_ok <- function(url){  message(url)  ok(url, verb = "get", useragent = "Maëlle Salmon checking links")}unique_urls <- unique_urls %>%  dplyr::group_by(https) %>%  dplyr::summarise(ok = get_ok(https))http <- dplyr::left_join(http, unique_urls, by = "https")httpsok <- dplyr::filter(http, ok)modify_url <- function(index, df = httpsok) {  row <- df[index,]  readLines(row$filepath) %>%    stringr::str_replace_all(row$url, row$https) %>%    writeLines(row$filepath)  }purrr::walk(seq_len(nrow(httpsok)), modify_url)
```
:::

You can [browse the
relatedPR](https://github.com/ropensci/roweb2/pull/582/). Note that in
theabove, we could have
used[`urltools`](https://github.com/Ironholds/urltools/) to parse URLs
andextract their scheme (http or https).

### docs.ropensci.org

To replace some ropensci.github.io links with docs.ropensci.org links,we
used the brute force approach below (there were only about 80
suchlinks).

::: {.highlight}
``` {.chroma tabindex="0"}
dotgithub <- dplyr::filter(all_urls, urltools::domain(url) == "ropensci.github.io")make_docs_url <- function(url, ropensci_pkgs = ropensci_pkgs) {  message(url)  newurl <- url  urltools::domain(newurl) <- "docs.ropensci.org"    if (crul::ok(newurl, verb = "get", useragent = "Maëlle Salmon checking links")) {    return(newurl)  } else {    return(url)  }}dotgithub <- dotgithub %>%  dplyr::group_by(url) %>%  dplyr::mutate(newurl = make_docs_url(url))modify_url <- function(index, df = dotgithub) {  row <- df[index,]  readLines(row$filepath) %>%    stringr::str_replace_all(row$url, row$newurl) %>%    writeLines(row$filepath)  }purrr::walk(seq_len(nrow(dotgithub)), modify_url)
```
:::

## Conclusion

In this tech note we saw how to use a combination of regularexpressions,
commonmark, xml2 and crul to identify links to be fixed inMarkdown
content. For html content, check out [the experimental checkerpackage by
François Michonneau](https://github.com/fmichonneau/checker).For
packages, have a look at [Bob Rudis'
RStudioadd-in](https://github.com/hrbrmstr/hrbraddins#whats-in-the-tin).

Some of the issues we fixed, like using relative rather than
absolutelinks, and not storing shortlinks, could be avoided in the
future bystricter URL guidelines for new content. We also plan to stop
usingClick here links, refer to [this page about why Click here links
arebad](https://webaccess.berkeley.edu/ask-pecan/click-here).

Now, a remaining issue is the frequency at which URL cleaning
shouldoccur. In [our dev guide](https://devguide.ropensci.org/), we
cleanlinks [before
eachrelease](https://devguide.ropensci.org/bookreleaseissue.html#bookreleaseissue),but
this website has no such schedule, so let's hope we remember toclean
URLs once in a while. Maybe some old pages could also be ["archived"
like this example](https://kb.iu.edu/d/abwp). When do *you* clean URLs
in your content,and how?


Last year we reported on the joy of using commonmark and xml2 to parseMarkdown content, like the source of this website built withHugo, in particular to extractlinks,at the time merely to count them. How about we go a bit further and usethe same approach to find links to be fixed? In this tech note we shallreport our experience using R to find broken/suboptimal links and fixthem. What is a bad URL?

We cleaned our website URLs with R!

Remember our recent post showing that one can wrangle Markdown filesprogrammatically without regex? Thattech note showed how to convert Markdown bodies to XML in order toextract information from them.

tinkr: editing Markdown documents using XML tools

A while ago weonboarded anexciting package, codemetarby Carl Boettiger. codemetar is an R specificinformation collector and parser for the CodeMetaproject. In particular, codemetar candigest metadata about an R package in order to fill the termsrecognized by CodeMeta. This meansextracting information from DESCRIPTION but also from e.g. continuousintegration
<sup>
 1
</sup>
badges in the README!

All the Badges One Can Earn: Parsing Badges of CRAN Packages READMEs

You might have read [my blog post analyzing the social weather
ofrOpenScionboarding](/blog/2018/05/10/onboarding-social-weather/),based
on a text analysis of GitHub issues. I extracted text out
ofMarkdown-formatted threads with regular expressions. I
basicallyhammered away at the issues using tools I was familiar with
until itworked! Now I know there's a much better and cleaner way, that
I'llpresent in this note. Read on if you want to extract insights
abouttext, code, links, etc. from R Markdown reports, Hugo website
sources,GitHub issues... without writing messy and smelly code!

## Introduction to Markdown rendering and parsing

This note will appear to you, dear reader, as an html page, either
hereon ropensci.org or on R-Bloggers, but I'm writing it as an R
Markdowndocument, using Markdown syntax. I'll knit it to Markdown and
thenHugo's Markdown
processor,[Blackfriday](https://github.com/russross/blackfriday), will
transformit to html. Elements such as `# blabla` thus get transformed
to`<h1 id="blabla">blabla</h1>`. Awesome!

The rendering of Markdown to html or XML can also be used as a way
to*parse* it, [which is what the `spelling` package does in order
toidentify
textsegments](https://github.com/ropensci/spelling/blob/622fc9cc200b69b3859f272c99c13c575dcb5105/R/parse-markdown.R#L12)of
R Markdown files, before spell checking them only, not code. I had anaha
moment when seeing this `spelling` strategy: why did I ever useregex to
parse Markdown for text analysis?! Transforming it to XMLfirst, and then
using XPath, would be much cleaner!

As a side-note, realizing how to simplify my old code made me think
of[Jenny Bryan's inspiring useR! keynote talk about
codesmells](https://github.com/jennybc/code-smells-and-feels). I asked
herwhether code full of regular expressions instead of dedicated
parsingtools was a code smell, sadly it doesn't have a specific name,
but sheconfirmed my feeling that *not* using dedicated purpose-built
toolsmight mean you'll end up "re-inventing all of that logic yourself,
inhacky way.". If you have code falling under the definition below,
maybetry to re-factor and if needed
[gethelp](https://masalmon.eu/2018/07/22/wheretogethelp/).

> It's that feeling when you want to do something that sounds simple but
> instead your code is like 10 stack overflow snippets slapped together
> that you could never explain to another human what they do 😰
> [pic.twitter.com/IF53AX6QvC](https://t.co/IF53AX6QvC)
>
> --- Alison Hill (@apreshill) [August 31,
> 2018](https://twitter.com/apreshill/status/1035526182392000514?ref_src=twsrc%5Etfw)

## From Markdown to XML

In this note I'll use my local fork of rOpenSci's website source, anduse
all the Markdown sources of blog posts as example data. The chunkbelow
is therefore not portable, sorry about that.

::: {.highlight}
``` {.chroma tabindex="0"}
roblog <- "C:\\Users\\Maelle\\Documents\\ropensci\\roweb2\\content\\blog"all_posts <- fs::dir_ls(roblog, regexp = "*.md")all_posts <- all_posts[all_posts != "_index.md"]
```
:::

My fork master branch isn't entirely synced. It has 202 posts.

The code below uses the
[`commonmark`package](https://github.com/jeroen/commonmark) to render
Markdown toXML. Commonmark is a standardized specification for Markdown
syntax by[John McFarlane](https://johnmacfarlane.net/tools). The
`commonmark` Rpackage by Jeroen Ooms wraps the
official[cmark](https://github.com/commonmark/cmark) library and is used
bye.g. GitHub to render issues and readmes. Note that my function
stillhas a hacky element, it uses a `blogdown` unexported function to
stripthe YAML header of posts! If you know a better way [feel free to
answermy question over at RStudio community
discussionforum](https://community.rstudio.com/t/does-blogdown-split-yaml-body-exist-as-an-exported-function-how-to-remove-the-yaml-of-a-markdown-r-markdown-document/13350/2).

::: {.highlight}
``` {.chroma tabindex="0"}
library("magrittr")get_one_xml <- function(md){  md %>%    readLines(encoding = "UTF-8") %>%    blogdown:::split_yaml_body() %>%    .$body %>%    commonmark::markdown_xml(extensions = TRUE) %>%    xml2::read_xml()}
```
:::

See what it gives me for one post.

::: {.highlight}
``` {.chroma tabindex="0"}
get_one_xml(all_posts[42])
```
:::

    ## {xml_document}## <document xmlns="http://commonmark.org/xml/1.0">##  [1] <paragraph>\n  <text>We just released a new version of </text>\n  < ...##  [2] <heading level="2">\n  <text>First, install and load taxize</text>\ ...##  [3] <code_block info="r">install.packages("rgbif")\n</code_block>##  [4] <code_block info="r">library(taxize)\n</code_block>##  [5] <heading level="2">\n  <text>New things</text>\n</heading>##  [6] <heading level="3">\n  <text>New functions: class2tree</text>\n</he ...##  [7] <paragraph>\n  <text>Sometimes you just want to have a visual of th ...##  [8] <paragraph>\n  <text>Define a species list</text>\n</paragraph>##  [9] <code_block info="r">spnames &lt;- c("Latania lontaroides", "Randia ...## [10] <paragraph>\n  <text>Then collect taxonomic hierarchies for each ta ...## [11] <code_block info="r">out &lt;- classification(spnames, db = "ncbi", ...## [12] <paragraph>\n  <text>Use </text>\n  <code>class2tree</code>\n  <tex ...## [13] <code_block info="r">tr &lt;- class2tree(out)\nplot(tr, no.margin = ...## [14] <paragraph>\n  <image destination="/assets/blog-images/2014-02-19-t ...## [15] <heading level="3">\n  <text>New functions: get_gbfid</text>\n</hea ...## [16] <paragraph>\n  <text>The Global Biodiversity Information Facility ( ...## [17] <paragraph>\n  <text>We added a similar function to our </text>\n   ...## [18] <code_block info="r">get_gbifid(sciname = "Poa annua", verbose = FA ...## [19] <code_block>##         1\n## "2704179"\n## attr(,"class")\n## [1] " ...## [20] <code_block info="r">get_gbifid(sciname = "Pinus contorta", verbose ...## ...

Headings, code blocks... all properly delimited and one XPath query
awayfrom us! Let me convert all posts before diving into parsing
examples.

::: {.highlight}
``` {.chroma tabindex="0"}
all_posts %>%  purrr::map(get_one_xml) -> blog_xml
```
:::

## Parsing the XML

### URLs parsing

Let's say I want to find out which domains are the most often linkedfrom
rOpenSci's blog. No need for any regular expression thanks
to`commonmark`, [`xml2`](https://xml2.r-lib.org/) and `urltools`!

::: {.highlight}
``` {.chroma tabindex="0"}
get_urls <- function(post_xml){  post_xml %>%    xml2::xml_find_all(xpath = './/d1:link', xml2::xml_ns(post_xml)) %>%    xml2::xml_attr("destination") %>%    urltools::url_parse()}# URLsblog_xml %>%  purrr::map_df(get_urls) %>%  dplyr::count(domain, sort = TRUE) %>%  head(n = 10) %>%  knitr::kable()
```
:::

| domain                               |    n |
|:-------------------------------------|-----:|
| github.com                           | 1111 |
| ropensci.org                         |  272 |
| twitter.com                          |  167 |
| cran.r-project.org                   |  130 |
| en.wikipedia.org                     |   60 |
| ropensci.github.io                   |   29 |
| doi.org                              |   27 |
| bioconductor.org                     |   15 |
| unconf17.ropensci.org                |   15 |
| [www.gbif.org](https://www.gbif.org) |   15 |

More Twitter than CRAN! We probably could do with less own-domain
usesince `/` would get us here too.

### R code parsing

Remember [that cool post by Matt Dancho analyzing David
Robinson'scode](https://www.business-science.io/learning-r/2018/03/03/how_to_learn_R_pt1.html)?In
theory you could clone any of your favorite blogs ([David
Robinson'sblog](https://github.com/dgrtwo/dgrtwo.github.com), [Julia
Silge'sblog](https://github.com/juliasilge/blog_by_hugo), etc.) to
analyzethem, no need to even webscrape first! Note [that you can git
clone fromR using the `git2r`
package](https://github.com/ropensci/git2r/).

::: {.highlight}
``` {.chroma tabindex="0"}
get_functions <- function(post_xml){  post_xml %>%    # select all code chunks    xml2::xml_find_all(xpath = './/d1:code_block', xml2::xml_ns(.)) %>%    # select chunks with language info    .[xml2::xml_has_attr(., "info")] %>%    # select R chunks    .[xml2::xml_attr(., "info") == "r"] %>%    # get the content of these chunks    xml2::xml_text() %>%    glue::glue_collapse(sep = "\n") -> code_text    # Base R code parsing tools  parsed_code <- try(parse(text = code_text,        keep.source = TRUE) %>%    utils::getParseData(),    silent = TRUE)    if(is(parsed_code, "try-error")){    # this happens because of output sometimes    # stored in R chunks when not using R Markdown    return(NULL)  }    if(is.null(parsed_code)){    return(NULL)  }      dplyr::filter(parsed_code,                grepl("FUNCTION", token))    }blog_xml %>%  purrr::map_df(get_functions) %>%  dplyr::count(text, sort = TRUE) %>%  head(n = 10) %>%  knitr::kable()
```
:::

| text             |   n |
|:-----------------|----:|
| library          | 263 |
| c                | 210 |
| aes              | 106 |
| filter           |  71 |
| mutate           |  64 |
| ggplot           |  58 |
| function         |  53 |
| install.packages |  50 |
| install_github   |  38 |
| select           |  38 |

Function definititions (`function`), basic stuff (`c`, `library`)
andtidyverse functions seem to be the most popular on the blog!

### Text parsing

After complementing our `commonmark`-`xml2` combo with `urltools`
andwith R base code parsing facilities... let's pair it
with[`tidytext`](https://www.tidytextmining.com/)! What are the words
mostcommonly used in rOpenSci's blog posts?

::: {.highlight}
``` {.chroma tabindex="0"}
get_text <- function(post_xml){  xml2::xml_find_all(post_xml,                     xpath = './/d1:text', xml2::xml_ns(post_xml)) %>%    xml2::xml_text(trim = TRUE) %>%    glue::glue_collapse(sep = " ") %>%    as.character() -> text    tibble::tibble(text = text)}blog_xml %>%  purrr::map_df(get_text) %>%  tidytext::unnest_tokens(word, text, token = "words") %>%  dplyr::filter(!word %in% tidytext::stop_words$word) %>%  dplyr::count(word, sort = TRUE) %>%  head(n = 10) %>%  knitr::kable()
```
:::

| word      |    n |
|:----------|-----:|
| data      | 1969 |
| package   | 1097 |
| ropensci  |  569 |
| packages  |  486 |
| time      |  412 |
| community |  394 |
| code      |  377 |
| github    |  358 |
| software  |  302 |
| science   |  297 |

This beats my old code! There's really something to be said
forpurpose-built tools.

## Conclusion

I hope this note will inspire you to use `commonmark` and `xml2`
whenanalyzing Markdown files. As mentioned earlier, Hugo or Jekyll
websitesources are Markdown files and GitHub issue threads are too so it
shouldopen up quite a lot of data! If you're new to XPath, [I'd
recommendreading
thisintroduction](https://www.w3schools.com/xml/xpath_intro.asp).
Theresults of XML-parsing are also better parsed without (your
writing)regular expressions: I have shown `urltools` for URL parsing,
that baseR has code parsing tools (`parse`, `getParsedData`), and I've
used`tidytext`.

Note that if you're into blog analysis, don't forget you can also
getinformation out of the YAML header using... [the
`yaml`package](https://github.com/viking/r-yaml), not regular
expressions!

As a bonus, maybe seeing this wrangling inspired you to *modify*Markdown
files programmatically? E.g. what if I wanted to automaticallyreplace
all level 1 headers with level 2 headers? We're working on that,stay
tuned [and if you want follow this
GitHubthread](https://github.com/commonmark/cmark/issues/264#issuecomment-417960358)!

Thanks to [Jeroen Ooms](https://github.com/jeroen),
[JennyBryan](https://jennybryan.org/) and
[JimHester](https://www.jimhester.com/) for their answering my XML
parsing(meta) questions.


You might have read my blog post analyzing the social weather ofrOpenScionboarding,based on a text analysis of GitHub issues. I extracted text out ofMarkdown-formatted threads with regular expressions. I basicallyhammered away at the issues using tools I was familiar with until itworked! Now I know there’s a much better and cleaner way, that I’llpresent in this note. Read on if you want to extract insights abouttext, code, links, etc.

Messaggi di Rogue Scholar

We cleaned our website URLs with R!

tinkr: editing Markdown documents using XML tools

All the Badges One Can Earn: Parsing Badges of CRAN Packages READMEs

In praise of Commonmark: wrangle (R)Markdown files without regex