Nicholas Potter

The United States Deparment of Agriculture National
AgriculturalStatistics Service ([USDA-NASS](https://www.nass.usda.gov/))
provides a wide range of agriculturaldata that includes animal, crop,
demographic, economic, andenvironmental measures across a number of
geographies and time periods.This data is available by direct download
or queriable via the[Quick Stats](https://quickstats.nass.usda.gov/)
interface. Whilethe Quick Stats tool puts a large amount of data into
the handsof users, the interface can be frustrating, especially when
tryingto access more than 50,000 records or hoping to automate
downloadingdata when new data is released. I
developed[rnassqs](https://ropensci.github.io/rnassqs/) as a solution
tothese frustrations. rnassqs provides a simple R interface forthe Quick
Stats API. By iterating over a set of parameters,R users can make
processing NASS data reproducible and automated.

### What is rnassqs and what can it do for you?

Succinctly, rnassqs will let you write an R script to downloaddata from
the Quick Stats database. Accessing the Quick Stats APIrequires a key,
which can be requested here:

<https://quickstats.nass.usda.gov/api>

The rnassqs package can be installed from CRAN or from GitHub:

::: {.highlight}
``` {.chroma tabindex="0"}
# Install from CRANinstall.packages("rnassqs")# Install from githubdevtools::install_github("ropensci/rnassqs")# Add the API keynassqs_auth("<your api key>") # just for this session# To add the API key permanently, add#     NASSQS_TOKEN="<your api key>"# to your .Renviron file, which can be accessed via#     usethis::edit_r_environ()
```
:::

Data can then be downloaded by selecting query parameters. For
example,the following downloads corn yields from 2017 onward for
counties inVirginia and Pennsylvania

::: {.highlight}
``` {.chroma tabindex="0"}
df <- nassqs(commodity_desc = "CORN",             year__GE = 2017,             agg_level_desc = "COUNTY",             state_alpha = c("VA", "PA"),             statisticcat_desc = "YIELD")
```
:::

### A real world use case

In a current working paper with coauthors Michael Brady and
KirtiRajagopalan, all of us at Washington State University, I
investigatethe relationship between climate and irrigated agriculture in
thewestern United States. USDA-NASS data is a crucial component ofthe
analysis, but can be complex to understand and download. Forexample, the
Census of Agriculture asks farmers to report thenumber of acres in three
different categories:

- land use (crop, pasture, wood, or other)
- harvested status (harvested or not)
- irrigation status (irrigated or not)

However, the census does not ask for acres under the full combinationof
land uses, so they must be reconstructed to the extent possible.This can
be done manually, but is tedious and subject to possibleerrors. Instead,
the full set of acres reported can be downloadedusing rnassqs. A good
strategy is to define the parameters for a setof queries that won't
change, and then add the specific parametersiteratively. In this case,
we can iterate over the type of acres butkeep the set of geographies and
years the same for each query. Tokeep data sizes small, we focus on
Yakima County in Washington State.

::: {.highlight}
``` {.chroma tabindex="0"}
acre_params <- list(  sector_desc = "ECONOMICS",  commodity_desc = "AG LAND",  agg_level_desc = "COUNTY",  unit_desc = "ACRES",  domain_desc = "TOTAL",  domaincat_desc = "NOT SPECIFIED",  county_name = "Yakima",  state_alpha = "WA",  year__GE = 1997,   year__LE = 2017)
```
:::

Then, specifying the specific parameters as a list to be iterated over:

::: {.highlight}
``` {.chroma tabindex="0"}
var_params <- list(  cropland = list(    class_desc = "CROPLAND",    prodn_practice_desc = "ALL PRODUCTION PRACTICES"  ),  pastureland = list(    class_desc = "PASTURELAND, (EXCL CROPLAND & WOODLAND)",    prodn_practice_desc = "ALL PRODUCTION PRACTICES"  ),  woodland = list(    class_desc = "WOODLAND",    prodn_practice_desc = "ALL PRODUCTION PRACTICES"  ),  other = list(    class_desc = "(EXCL CROPLAND & PASTURELAND & WOODLAND)",    prodn_practice_desc = "ALL PRODUCTION PRACTICES"  ),  cropland_harvested = list(    class_desc = "CROPLAND, HARVESTED",    prodn_practice_desc = "ALL PRODUCTION PRACTICES"  ),  cropland_harvested_irrigated = list(    class_desc = "CROPLAND, HARVESTED",    prodn_practice_desc = "IRRIGATED"  ),  irrigated = list(    class_desc = "ALL CLASSES",    prodn_practice_desc = "IRRIGATED"  ),  irrigated_excl_cropland_harvested = list(    class_desc = "(EXCL HARVESTED CROPLAND)",    prodn_practice_desc = "IRRIGATED"  ))
```
:::

Now to actually download the data:

::: {.highlight}
``` {.chroma tabindex="0"}
data_list <- lapply(names(var_params), function(v) {  # create a new parameter list from the base list  params <- acre_params  # Assign parameters for the specific variable in v  vp <- var_params[[v]]  for(p in names(vp)) { params[[p]] <- vp[[p]] }  # Download the data  d <- nassqs(params)  remove(params)  # Assign a variable for ease of tracking  d$variable <- v  # Add the data.frame to the list of data  d})# Convert the list of data.frames to one data.framedf <- dplyr::bind_rows(data_list) # with 'dplyr'#df <- data.table::rbindlist(data_list) # with 'data.table'#df <- do.call(rbind, data_list) # with base R
```
:::

We can present a compact table of acres using tidyr:

::: {.highlight}
``` {.chroma tabindex="0"}
library(dplyr)library(tidyr)df %>%  select(variable, year, Value) %>%  spread(key = year, value = Value)
```
:::

which generates a `data.frame` that looks like:

<figure>
<img src="/img/blog-images/2019-11-26-rnassqs/df1a.png" width="800"
alt="Output of data.frame without converting to numeric." />
</figure>

An important note: `nassqs()` retains the `character` typeof the `Value`
column because there are often missing valuecodes returned. Here we can
see that `cropland` and `woodland`have missing codes for 1997. Before
doing any calculationswe want to convert the values to numeric, but also
take noteof missing codes to help decide how to handle missing values.In
this case, we convert them to `NA` and can convert values tonumeric with

::: {.highlight}
``` {.chroma tabindex="0"}
df %>%  select(variable, year, Value) %>%  transmute(    variable,    year,    value = as.numeric(gsub(",", "", Value))) %>%  spread(key = year, value = value)
```
:::

which generates a data.frame with values converted to numeric:

<figure>
<img src="/img/blog-images/2019-11-26-rnassqs/df2a.png" width="800"
alt="Output of data.frame converted to numeric." />
</figure>

### Developing and submitting your first R package to rOpenSci

I began development on rnassqs in the fall of 2017 when I wasconfronted
with having to manually download many different variablesfor a number of
different counties and years. Having never developedan R package before,
I found great use in two sources:

- Hadley's [R Packages](http://r-pkgs.had.co.nz/)
- The httr package's vignette [Best practices for API
  Packages](https://cran.r-project.org/web/packages/httr/vignettes/api-packages.html)

After a year or so of unofficial releases and work,[Hao
Ye](https://twitter.com/Hao_and_Y) suggested I submit the packageto
rOpenSci. It's hard to overstate the value of having two friendlyand
thorough reviewers ([Adam Sparks](https://github.com/adamhsparks)and
[Neal Richardson](https://github.com/nealrichardson)) pour overyour
package in detail. Their comments were extremely helpful (hereis
the[review
thread](https://github.com/ropensci/software-review/issues/298))and
greatly improved the package in a number of ways:

- **Testing**: I initially had poorly organized and confusing
  tests,largely because I had little experience. Luckily one Neal
  Richardsonwrote the
  [httptest](https://github.com/nealrichardson/httptest) packagethat
  makes it easier to make mock API calls to test without making the
  actualcall to the API service, which becomes important for passing
  CRAN checks.
- **Documentation**: As the author of functions, it's hard to
  documentfrom the perspective of someone who is new to the function.
  The reviewsprovided that perspective and made the documentation much
  clearer.
- **Simplicity**: The rnassqs code base was unecessarily complicatedwhen
  I initially submitted it. It was much cleaner, easier to
  understand,and easier to maintain by the time the review was finished.

In addition, a benefit I consider on par with the above improvements
tornassqs was the experience of interacting with reviewers. Both
werefriendly and helpful and contributed time and energy far beyond what
Iexpected. Their example served as a strong cultural norm to guide my
workas a reviewer for a later package.

### In closing

In summary, there are two lessons in particular I hope to retain for
futurework:

1.  On the scale of simple but rigid to flexible but complex, I tend
    tolean toward the latter too early, trying to abstract and
    complicate to avoidrepeating myself. I think my work would be
    cleaner, faster, and more coherentif I kept a more narrow focus and
    thought at a meta level about package designbefore diving in to
    write functions that encapsulate all possible use functions.
2.  The benefit of other perspectives cannot be overstated. Getting
    feedbackearlier could have avoided unecessarily complicating the
    package code.

If you're interested in learning more about rnassqs, the[package
documentation](https://ropensci.github.io/rnassqs/) providesan
introduction and guide to use as well as a[vignette with detailed
examples](https://docs.ropensci.org/rnassqs/articles/rnassqs.html).A
goal going forward is to improve the ease of developing queries, which
remainsdifficult, requiring an understanding of the peculiarities of the
Quick Statsdatabase. Some convenience functions like `nassqs_acres()`
and `nassqs_yields()`may help with that, but there must be better ways.
The rnassqs[repository on GitHub](https://github.com/ropensci/rnassqs)
is the best place tofile issues or feature requests, both of which are
very welcome.

### Acknowledgements

Thank you to rOpensci reviewers [Adam
Sparks](https://github.com/adamhsparks) and[Neal
Richardson](https://github.com/nealrichardson) for their efforts
andsuggestions which greatly improved the package. Thanks as well
to[contributors](https://docs.ropensci.org/rnassqs/authors.html) to
rnassqs,much appreciated!


The United States Deparment of Agriculture National AgriculturalStatistics Service (USDA-NASS) provides a wide range of agriculturaldata that includes animal, crop, demographic, economic, andenvironmental measures across a number of geographies and time periods.This data is available by direct download or queriable via theQuick Stats interface.

Messaggi di Rogue Scholar

rnassqs: accessing USDA agricultural data via API