Georgia State University

Andrew Heiss

In a project I’m working on, I need to generate predictions from a logistic regression model.

Manually generate predicted values for logistic regression with matrix multiplication in R

Who Cares about Crackdowns? Exploring the Role of Trust in Individual Philanthropy

Causal Inference in Conjoint Analysis: Understanding Multidimensional Choices via Stated Preference Experiments

Measuring Subgroup Preferences in Conjoint Experiments

Contingent Citizens

Religion of a Different Color

The Ties That Double Bind: Social Roles and Women's Underrepresentation in Politics

In my research, I study international nongovernmental organizations (INGOs) and look at how lots of different institutional and organizational factors influence INGO behavior. For instance, many authoritarian regimes have passed anti-NGO laws and engaged in other forms of legal crackdown, which has forced NGOs to change their programming strategies and their sources of funding.

The ultimate practical guide to conjoint analysis with R

As I explained in my previous blog post, in June 2023 I drove my family across the country in a 5,000-mile roundtrip odyssey from Atlanta, Georgia to just-south-of-Salt Lake City, Utah, and back again.

Road trip analysis! How to use and play with Google Location History in R

I’ve taught a course on data visualization with R since 2017, and it’s become one of my more popular classes, especially since it’s all available asynchronously online with hours of Creative Commons-licensed videos and materials.

Making Middle Earth maps with R

As a field, statistics is really bad at naming things. Take, for instance, the term “fixed effects.” In econometrics and other social science-flavored statistics, this typically refers to categorical terms in a regression model.

Marginal and conditional effects for GLMMs with {marginaleffects}

Downloadable cheat sheets! You can download PDF, SVG, and PNG versions of the diagrams and cheat sheets in this post, as well as the original Adobe Illustrator and InDesign files, at the bottom of this post  Do whatever you want with them!

Visualizing the differences between Bayesian posterior predictions, linear predictions, and the expectation of posterior predictions

Diagrams! You can download PDF, SVG, and PNG versions of the marginal effects diagrams in this guide, as well as the original Adobe Illustrator file, here: PDFs, SVGs, and PNGs Illustrator .ai file  Do whatever you want with them! They’re licensed under Creative Commons Attribution-ShareAlike (BY-SA 4.0).  I’m a huge fan of doing research and analysis in public.

Marginalia: A guide to figuring out what the heck marginal effects, marginal slopes, average marginal effects, marginal effects at the mean, and all these other marginal things are

Uncertainty in the design stage of two‐stage Bayesian propensity score analysis

Read the previous post first! This post is a sequel to the previous one on Bayesian propensity scores and won’t make a lot of sense without reading that one first. Read that one first! In my previous post about how to create Bayesian propensity scores and how to legally use them in a second stage outcome model, I ended up using frequentist models for the outcome stage.

How to create a(n almost) fully Bayesian outcome model with inverse probability weights

Discussion of “On Bayesian Estimation of Marginal Structural Models”

Multiple Imputation for Nonresponse in Surveys

On Bayesian Estimation of Marginal Structural Models

The Central Role of Bayes’ Theorem for Joint Estimation of Causal Effects and Propensity Scores

This post combines two of my long-standing interests: causal inference and Bayesian statistics. I’ve been teaching a course on program evaluation and causal inference for a couple years now and it has become one of my favorite classes ever.

How to use Bayesian propensity scores and inverse probability weights

In most of my research, I work with country-level panel data where each row is a country in a specific year (Afghanistan in 2010, Afghanistan in 2011, and so on), also known as time-series cross-sectional (TSCS) data.

A guide to working with country-year panel data and Bayesian multilevel models

The design of ballantines

Visualizing Multiple Regression

Regression is the core of my statistics and program evaluation/causal inference courses. As I’ve taught different stats classes, I’ve found that one of the regression diagnostic statistics that students really glom onto is . Unlike lots of regression diagnostics like AIC, BIC, and the joint F-statistic,  has a really intuitive interpretation—it’s the percent of variation in the outcome variable explained by all the explanatory variables.

Exploring R² and regression variance with Euler/Venn diagrams

At the end of my previous post on beta and zero-inflated-beta regression, I included an example of a multilevel model that predicted the proportion of women members of parliament based on whether a country implements gender-based quotas for their legislatures, along with a few different control variables. I also included random effects for year and region in order to capture time- and geography-specific trends.

A guide to correctly calculating posterior predictions and average marginal effects with multilievel Bayesian models

How to Make Causal Inferences with Time-Series Cross-Sectional Data under Selection on Observables

A Primer on Inverse Probability of Treatment Weighting and Marginal Structural Models

Since my last two blog posts on binary and continuous inverse probability weights (IPWs) and marginal structural models (MSMs) for time-series cross-sectional (TSCS) panel data, I’ve spent a ton of time trying to figure out why I couldn’t recover the exact causal effect I had built in to those examples when using panel data. It was a mystery, and it took weeks to figure out what was happening.

Marginal structural models for panel data with GEE and multilevel models

<em>
 This is written for instructors in the Department of Public Management and Policy at the Andrew Young School of Policy Studies at Georgia State University, but it’s hopefully widely applicable too.
</em>
With more than 100 universities moving their teaching online (including Emory just last night), it’s looking more and more inevitable that GSU will make a similar switch any time now.

Emergency online teaching resources

Now that I’m on the tenure track, I’ve been looking for a way to keep track of my different research projects so I can get them all finished and published. Matt Lebo’s “Managing Your Research Pipeline” presents a neat way of quantifying and tracking the progress of your research, and I recently adopted it for my own stuff. I even made a fancy R Markdown +
<strong>
 flexdashboard
</strong>
dashboard to show the status of the pipeline interactively.

Create a dynamic dashboard with R, flexdashboard, and Shiny

(See this notebook on GitHub)  A year ago, I wrote about how to use R to solve a typical microeconomics problem: finding the optimal price and quantity of some product given its demand and cost. Doing this involves setting the first derivatives of two functions equal to each other and using algebra to find where they cross.

Chidi’s budget and utility: doing algebra and calculus with R and yacas

I am
<em>
 so beyond thrilled
</em>
to announce that I’ll be joining the Andrew Young School of Policy Studies at Georgia State University in Fall 2019 as an assistant professor in the Department of Public Management and Policy. I’ll be teaching classes in statistics/data science, economics, and nonprofit management in beautiful downtown Atlanta, and we’ll be moving back to the South. I am so so excited about this!

The academic job search finally comes to an end

Complete tutorial and code  See the full tutorial here. You can also see the tutorial’s code here and the code for the final API here.

DIY API with Make and {plumber}

Downloads  Jump to the downloads and get your own free pattern and template files! In the data science world, hex stickers with logos of developers’ favorite packages are all the rage.

Hex sticker/logo cross stitch pattern

The students in my summer data visualization class are finishing up their final projects this week and I’ve been answering a bunch of questions on our class Slack. Often these are relatively standard reminders of how to tinker with specific ggplot layers (chaning the colors of a legend, adding line breaks in labels, etc.), but today one student had a fascinating and tricky question that led me down a realy fun dataviz rabbit hole.

How to fill maps with density gradients with R, {ggplot2}, and {sf}

Introduction to Causal Inference

I’ve been teaching a course on program evaluation since Fall 2019, and while part of the class is focused on logic models and the more managerial aspects of evaluation, the bulk of the class is focused on causal inference. Ever since reading Judea Pearl’s
<em>
 The Book of Why
</em>
in 2019, I’ve thrown myself into the world of DAGs, econometrics, and general causal inference, and I’ve been both teaching it and using it in research ever since.

Do-calculus adventures! Exploring the three rules of do-calculus in plain language and deriving the backdoor adjustment formula by hand

The causal interpretation of estimated associations in regression models

Rage Against the Mean – A Review of Distributional Regression Approaches

The Global Impact of Quotas

In the data I work with, it’s really common to come across data that’s measured as proportions: the percent of women in the public sector workforce, the amount of foreign aid a country receives as a percent of its GDP, the percent of religious organizations in a state’s nonprofit sector, and so on.  When working with this kind of data as an outcome variable (or dependent variable) in a model, analysis gets tricky if you use standard models like

A guide to modeling proportions with Bayesian beta and zero-inflated beta regression models

See this notebook on GitHub. You can (and should) download the project from there if you want to follow along and try this out. tl;dr: Skip to the completed example. I use
<strong>
 blogdown
</strong>
to generate the websites for all the courses I teach, and it’s delightful to not have to worry about databases and server configurations.

Automatically zip up subdirectories with Make

tl;dr  If you want to skip the explanation and justification for why you might want separate bibliographies, you can skip down to the example section, or just go see some example files at GitHub. Why use separate bibliographies? In academic articles, it’s common to have a supplemental appendix with extra tables, figures, robustness checks, additional math, proofs, and other details.

How to create separate bibliographies in a Quarto document

Ordered Beta Regression: A Parsimonious, Well-Fitting Model for Continuous Data with Lower and Upper Bounds

I’ve been finishing up a project that uses ordered Beta regression (Kubinec 2022), a neat combination of Beta regression and ordered logistic regression that you can use for modeling continuous outcomes that are bounded on either side (in my project, we’re modeling a variable that can only be between 1 and 32, for instance). It’s possible to use something like zero-one-inflated Beta regression for outcomes like this, but that kind of model

Guide to understanding the intuition behind the Dirichlet distribution

R For Marketing Research and Analytics

Data Analysis Using Regression and Multilevel/Hierarchical Models

City limits to partisan polarization in the American public

I recently posted a guide (mostly for future-me) about how to analyze conjoint survey data with R. I explore two different estimands that social scientists are interested in—causal average marginal component effects (AMCEs) and descriptive marginal means—and show how to find them with R, with both frequentist and Bayesian approaches. However, that post is a little wrong. It’s not
<em>
 wrong
</em>
wrong, but it is a bit oversimplified.

The ultimate practical guide to multilevel multinomial conjoint analysis with R

I’ve been working on converting a couple of my dissertation chapters into standalone articles, so I’ve been revisiting and improving my older R code. As part of my dissertation work, I ran a global survey of international NGOs to see how they adjust their programs and strategies when working under authoritarian legal restrictions in dictatorship.

A guide to Bayesian proportion tests with R and {brms}

Pandoc-flavored Markdown makes it really easy to cite and reference things. You can write something like this (assuming you use this references.bib BibTeX file): --- title: "Some title" bibliography: references.bib --- According to @Lovelace:1842, computers can calculate things.

One Simple Trick™ to create inline bibliography entries with Markdown and pandoc

In
<em>
 The Two Towers
</em>
, while talking with Eowyn, Aragorn casually mentions that he’s actually 87 years old. When Aragorn is off running for miles and miles and fighting orcs and trolls and Uruk-hai and doing all his other
<em>
 Lord of the Rings
</em>
adventures, he hardly behaves like a regular human 87-year-old. How old is he really?

How old was Aragorn in regular human years?

My longstanding workflow for writing, citing, and PDF management   When I started my first master’s degree program in 2008, I decided to stop using Word for all my academic writing and instead use plain text Markdown for everything. Markdown itself had been a thing for 4 years, and MultiMarkdown—a pandoc-like extension of Markdown that could handle BibTeX bibliographies—was brand new.

How to migrate from BibDesk to Zotero for pandoc-based writing

In one of the assignments for my data visualization class, I have students visualize the number of essential construction projects that were allowed to continue during New York City’s initial COVID shelter-in-place order in March and April 2020. It’s a good dataset to practice visualizing amounts and proportions and to practice with
<strong>
 dplyr
</strong>
’s group_by() and summarize() and shows some interesting trends.

Quick and easy ways to deal with long labels in ggplot2

Update #1  An update to knitr has made it a ton easier to embed fonts in SVG files from R. Jump to the update to see how. Update #2  Also, it’s possible to change TikZ fonts and not use Computer Modern for everything! Jump to the second update to see how.

How to automatically convert TikZ images to SVG (with fonts!) from knitr

How Much Should We Trust Staggered Difference-In-Differences Estimates?

Difference-in-Differences with multiple time periods

Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects

Difference-in-differences with variation in treatment timing

Estimating dynamic treatment effects in event studies with heterogeneous treatment effects

The world of econometrics has been roiled over the past couple years with a bunch of new papers showing how two-way fixed effects (TWFE; situations with nested levels of observations, like country-year, state-month, etc.) estimates of causal effects from difference-in-differences-based natural experiments can be biased when treatment is applied at different times.

Exploring Pamela Jakiela’s simple TWFE diagnostics with R

Downloads  Jump to the downloads and get your own free pattern and template files! Apparently I now only produce cross stitch content. Thanks, pandemic. In preparation for season 2 of the incredible
<em>
 Ted Lasso
</em>
, I made a cross stitch version of the AFC Richmond crest, and I’m really happy with how it turned out!

AFC Richmond / Ted Lasso cross stitch pattern

Downloads  Jump to the downloads and get your own free pattern! For my latest pandemic art medium (on this, the three hundred and nineteenth day of sheltering in place), I decided to teach myself how to cross stitch. I did a ton of needlepoint as a kid and teen, but was always afraid of cross stitch because it was so much smaller and more delicate.

Bayesian (cross stitch) sampler

Constructing Inverse Probability Weights for Marginal Structural Models

Causal Inference: What If (the book)

Robust Estimation of Inverse Probability Weights for Marginal Structural Models

Marginal Structural Models and Causal Inference in Epidemiology

In my post on generating inverse probability weights for both binary and continuous treatments, I mentioned that I’d eventually need to figure out how to deal with more complex data structures and causal models where treatments, outcomes, and confounders vary over time.

Generating inverse probability weights for marginal structural models with time-series cross-sectional panel data

Constructing Inverse Probability Weights for Continuous Exposures

My program evaluation class is basically a fun wrapper around topics in causal inference and econometrics. I’m a big fan of Judea Pearl-style “causal revolution” causal graphs (or DAGs), and they’ve made it easier for both me and my students to understand econometric approaches like diff-in-diff, regression discontinuity, and instrumental variables.

Generating inverse probability weights for both binary and continuous treatments

GSU uses Microsoft’s Office365 for e-mail, which is fine. My previous institutions—Duke and BYU—both use it too, and it’s pretty standard. GSU also enforces 2-factor authentication (2FA) with Duo, which is also fine. Everybody should use some sort of 2FA for all their important accounts! However, for whatever reason, GSU’s version of Duo’s 2FA doesn’t allow you to generate app-specific passwords for things like e-mail.

Convert Markdown to rich text (with syntax highlighting!) in any macOS app

(See this notebook on GitHub)  I’ve been teaching program evaluation in the MPA/MPP program at GSU for the past semester and a half, and since I was given free rein over how to teach it, I decided to make it as modern and cutting edge as possible.

Ways to close backdoors in DAGs

(See this notebook on GitHub)  Taking a sample from two groups from a population and seeing if there’s a significant or substantial difference between them is a standard task in statistics.

Half a dozen frequentist and Bayesian ways to measure the difference in means in two groups

Even though I’ve been teaching R and statistical programming since 2017, and despite the fact that I do all sorts of heavily quantitative research,
<em>
 I’m really really bad at probability math
</em>
.  Like super bad. The last time I truly had to do set theory and probability math was in my first PhD-level stats class in 2012.

Calculating birthday probabilities with R instead of math

A few days ago, my wife, a bunch of my kids, and I were huddled around a big wall map of the United States, joking about the relative unimportance of Rhode Island, the smallest state in the US. It’s one of the states I never ever think about:  …and it’s just so
<em>
 small
</em>
.  Amid the joking, my wife came to Rhode Island’s defense by declaring that even though it’s so small, it has one of the highest proportions of coastline to land borders.

Calculating the proportion of US state borders that are coastlines

Difference-in-Difference in the Time of Cholera: a Gentle Introduction for Epidemiologists

Choosing the Causal Estimand for Propensity Score Analysis of Observational Studies

R for Political Data Science

Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference

The Effect: An Introduction to Research Design and Causality | The Effect

Pearl before economists: the book of why and empirical economics

Counterfactuals and Causal Inference

In my causal inference class, I spend just one week talking about the Rubin causal model and potential outcomes.

Demystifying causal inference estimands: ATE, ATT, and ATU

I’ve used Garrick Aden-Buie’s tidyexplain animations since he first made them in 2018. They’re incredibly useful for teaching—being able to see which rows left_join() includes when merging two datasets, or which cells end up where when pivoting longer or pivoting wider is so valuable.

Visualizing {dplyr}’s mutate(), summarize(), group_by(), and ungroup() with animations

At the end of June 2024, Posit released a beta version of its next-generation IDE for data science: Positron. This follows Posit’s general vision for language-agnostic data analysis software: RStudio PBC renamed itself to Posit PBC in 2022 to help move away from a pure R focus, and Quarto is pan-lingual successor to R Markdown.

Fun with Positron

In a couple days, I’m going to drive across the country to Utah, my home state. I haven’t been out west with my whole family in four years—not since 2019 when we moved from Spanish Fork, Utah to Atlanta, Georgia. According to Google Maps, it’s a mere 1,900 miles (3,000 kilometers) through the middle of the United States and should take 28 hours, assuming no stops.

How to make fancy road trip maps with R and OpenStreetMap

I always forget how to deal with logged values in ggplot—particularly things that use the natural log.

How to use natural and base 10 log scales in ggplot2

In a research project I’ve been working on for several years now, we’re interested in the effect of anti-NGO legal crackdowns on various foreign aid-related outcomes: the amount of foreign aid a country receives and the proportion of that aid dedicated to contentious vs. non-contentious causes or issues. These outcome variables are easily measurable thanks to the AidData project, but they post a tricky methodological issue.

A guide to modeling outcomes that have lots of zeros with Bayesian hurdle lognormal and hurdle Gaussian regression models

Andrew Heiss's blog

::: {.callout .callout-style-default .callout-important .callout-titled}
::: {.callout-header .d-flex .align-content-center}
::: {.callout-icon-container}
:::

::: {.callout-title-container .flex-fill}
Complete tutorial and code
:::
:::

::: {.callout-body-container .callout-body}
[See the full tutorial here.](_book/){target="_blank" rel="noopener"}
You can also see [the tutorial's code
here](https://github.com/andrewheiss/diy-api-plumber-quarto-ojs) and the
[code for the final API
here](https://github.com/andrewheiss/basic-plumber-api).
:::
:::

For years, I've tracked all sorts of data about myself (and my family)
through Google Forms, [Airtable](https://www.airtable.com/), and devices
like Fitbits to keep track of all sorts of things: personal goals,
progress of research projects, current health status, books read, and so
on.

It's nice to have all this data, but it's hard to use it all
immediately. I often look at it at the end of the year, or every few
months, or whatever, but having an instant snapshot is helpful too.
That's why people invented data dashboards, after all.

I like R a lot, and R has the ability to make dashboards, like with
[{flexdashboard}](https://pkgs.rstudio.com/flexdashboard/) and
[Shiny](https://shiny.posit.co/). I've made several dashboards for
tracking things like health and reading and research---[I even have a
blog post about making one with
{flexdashboard}](https://www.andrewheiss.com/blog/2020/01/01/flexdashboard-dynamic-data/)!

But I've always run into issues with getting live data. With
{flexdashboard}, you can make it grab the most recent version of the
data you're interested in *when you knit the document*, but then to
update the graphs and tables in the document, you have to re-knit it.
With Shiny, there are ways to dynamically grab the latest data, but then
you have to run a whole Shiny server, and that's hard and costs money
*and* it's slow---it can sometimes take a few minutes to reanimate a
hibernating Shiny app!

However, nowadays it's possible to use [Observable
JS](https://quarto.org/docs/computations/ojs.html) chunks in Quarto that
automatically grab live data from the internet and display it, like
this:

::: {.cell}
Show the OJS code

::: {#cb1 .sourceCode .cell-code startfrom="41" source-offset="46" style="background: #f1f3f5;"}
```` {.sourceCode .java .code-with-copy}
```{ojs}
//| echo: fenced
//| code-fold: true
//| code-summary: "Show the OJS code"
d3 = require('d3')

viewof year_to_show = Inputs.radio(["2023", "2024"], {value: "2023", label: "Year to show"})

books = await d3.json(
  "https://api.andrewheiss.com/books_simple?year=" + year_to_show
)

book_noun = (books.count[0] === 1 ? " book read" : " books read")

Plot.plot({
  title: books.count[0] + book_noun + " in " + year_to_show,
  y: {
    label: "Books read",
    grid: false,
    percent: false
  },
  x: {
    label: "Month",
    domain: books.monthly_count.map(d => d.read_month_fct),
  },
  marks: [
    Plot.ruleY([0]),
    Plot.axisX({label: null, ticks: null}),
    Plot.axisY({label: null, ticks: null}),

    Plot.barY(books.monthly_count, {
      x: "read_month_fct", 
      y: "count", 
      fill: "#f3752f",
      tip: {
        format: {
          x: true,
          y: true
        }
      }
    })
  ]
})
```
````
:::

::: {.cell-output .cell-output-display}
<div>

::: {#ojs-cell-1-1 nodetype="declaration"}
:::

</div>
:::

::: {.cell-output .cell-output-display}
<div>

::: {#ojs-cell-1-2 nodetype="declaration"}
:::

</div>
:::

::: {.cell-output .cell-output-display}
<div>

::: {#ojs-cell-1-3 nodetype="declaration"}
:::

</div>
:::

::: {.cell-output .cell-output-display}
<div>

::: {#ojs-cell-1-4 nodetype="declaration"}
:::

</div>
:::

::: {.cell-output .cell-output-display}
<div>

::: {#ojs-cell-1-5 nodetype="expression"}
:::

</div>
:::
:::

And now that [Quarto supports
dashboards](https://quarto.org/docs/dashboards/), you can create entire
Shiny-like dashboards that can load and display data without needing a
Shiny server. [Like this one!](_book/dashboard/)

The trickiest part of all this, though, is getting data from all around
the internet (Google Sheets, Airtable databases, RSS feeds, Fitbit,
etc.) into an easily accessible, clean, and usable format that you can
feed into things like Observable plots or R.

Fortunately there's a good (and really neat!) solution for this! You can
use the [{plumber} R package](https://www.rplumber.io/) to create your
own API that you can use to grab and clean data from all around the
internet. And to simplify life, you can use other services like
[Make.com](https://www.make.com/en) to deal with the hard work of
regularly checking in on different parts of the internet (checking RSS
feeds, reading Google/Airtable data, logging into services like Fitbit).

In the end, you can have a server like `api.yourname.com` and access
JSON, CSV, or .rds data like
`api.yourname.com/books?start_date=2024-01-01`. You can then use that in
an R file, in a Python script, or in a Quarto document with Observable
JS. It's magical!

To explain and illustrate this whole process, I started out writing a
blog post, but it got long and complex, so I wrote a literal book
instead.

[Access it here for a full tutorial.](_book/){target="_blank"
rel="noopener"}

::: {.page-columns .page-full}
::: {.img-fluid .quarto-figure .quarto-figure-center .page-columns .page-full}
<figure class="figure page-columns page-full">
<p><img
src="https://www.andrewheiss.com/blog/2024/01/12/diy-api-plumber-quarto-ojs/_book/assets/out/overview.svg"
class="img-fluid figure-img column-page-inset" /></p>
<figcaption>Overview of the process</figcaption>
</figure>
:::
:::


::: {.callout .callout-style-default .callout-note .callout-titled}
::: {.callout-header .d-flex .align-content-center}
::: {.callout-icon-container}
:::

::: {.callout-title-container .flex-fill}
tl;dr
:::
:::

::: {.callout-body-container .callout-body}
If you want to skip the explanation and justification for why you might
want separate bibliographies, you can skip down to the example section,
or just [go see some example files at
GitHub](https://github.com/andrewheiss/multibib-testing).
:::
:::

::: {#why-use-separate-bibliographies .section .level2}
## Why use separate bibliographies? {#why-use-separate-bibliographies .anchored anchor-id="why-use-separate-bibliographies"}

In academic articles, it's common to have a supplemental appendix with
extra tables, figures, robustness checks, additional math, proofs, and
other details. Putting content in the appendix is important for
providing additional evidence for the paper's argument---and for
placating reviewers who want to see a dozen more robustness checks.
Also, journal articles have word count limits, and sticking stuff in the
appendix is a helpful way to circumvent those limits, since the appendix
doesn't get typeset with the rest of the article and is instead posted
to the journal's website as a supplemental file.

For years, my approach to making an appendix has been to use two
separate Markdown files (either plain Markdown, R Markdown, or Quarto
Markdown):

- `manuscript.md`: The actual paper
- `appendix.md`: The appendix

That's nice and straightforward and easy. I get two separate documents
in the end, [like this
project](https://stats.andrewheiss.com/cautioning-canary/):

::: {.img-border .img-fluid .quarto-figure .quarto-figure-center}
<figure class="figure">
<p><img
src="https://www.andrewheiss.com/blog/2023/12/11/separate-bibliographies-quarto/img/multiple-outputs.png"
class="img-border img-fluid figure-img" style="width:80.0%" /></p>
<figcaption>Multiple outputs for separate documents</figcaption>
</figure>
:::

Using separate files also lets me find the word count for the manuscript
without needing to account for the appendix (like with my [Quarto word
count extension](https://github.com/andrewheiss/quarto-wordcount)).

The biggest downside to this system, though, is that it's impossible to
reference the appendix from the manuscript. A magical feature of pandoc
(and Quarto in particular) is its ability to [cross
reference](https://quarto.org/docs/authoring/cross-references.html)
tables, figures, equations, sections, and other content in your
document. I can write something like this:

::: {#cb1 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .default .code-with-copy}
See @fig-results and @tbl-details for the results.
```
:::

...which turns into this when rendering the document:

> See Figure 1 and Table 1 for the results.

Those numbers are automatically incremented, so it might say "Figure 4"
if you have images earlier in the document. The numbers can have
hyperlinks added to them so you can jump to the corresponding figure or
table, and if you're using rendering to HTML with Quarto, you also get a
neat little preview popup when you hover over the link:

::: {.img-border .img-fluid .quarto-figure .quarto-figure-center}
<figure class="figure">
<p><img
src="https://www.andrewheiss.com/blog/2023/12/11/separate-bibliographies-quarto/img/hover-reference.png"
class="img-border img-fluid figure-img" style="width:80.0%" /></p>
<figcaption>Popup hover preview</figcaption>
</figure>
:::

However, you cannot cross reference content that is in a separate file.
For instance, if I'm writing in `manuscript.qmd` and I have a table in
`appendix.qmd` named `tbl-extra-details`, if I write this:

::: {#cb2 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .default .code-with-copy}
See @fig-results and @tbl-details for the results, and @tbl-extra-details for complete details.
```
:::

...I'll get this when rendering:

> See Figure 1 and Table 1 for the results, and **?tbl-extra-details**
> for complete details.

My only way of fixing this has been to manually type the number of the
appendix table:

::: {#cb3 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .default .code-with-copy}
See @fig-results and @tbl-details for the results, and Table A1 for complete details.
```
:::

...but this is annoying because (1) if I rearrange any of the content in
the appendix, that table might not be A1 anymore, and (2) there's no
neat automatic hyperlink to Table A1.

The manuscript and appendix documents can't see each other, so there's
no way to make them talk to each other when generating cross references.

Quarto has a good solution for this for longer documents: [the Quarto
book format](https://quarto.org/docs/books/). A Quarto book is a special
kind of website where you can maintain separate .qmd files (like
`chapter1.qmd`, `chapter2.qmd`, `appendix.qmd`, etc.), and when you
render, the documents *can* see each other, so you can refer to content
in other documents.

That's great for books, and it could theoretically work for shorter
documents like articles---make a single-chapter "book" where
`manuscript.qmd` is the first chapter and `appendix.qmd` is the
appendix. However, there are some issues when rendering to PDF---[see
"Approach 2" at this discussion
here](https://github.com/quarto-dev/quarto-cli/discussions/4581).

An easier solution would be to keep all the content in the same Markdown
file and put the appendix stuff at the end under a heading like
`# Appendix`. That way you don't have to worry about getting multiple
documents to see each other and cross-referencing works just fine.
Quarto even [has a neat
feature](https://quarto.org/docs/authoring/appendices.html) where it can
generate an end-of-the-document appendix for you. And my [Quarto word
count extension](https://github.com/andrewheiss/quarto-wordcount) has
the ability to count the words in the appendix separately from the rest
of the document.

[In one paper I have under
review](https://stats.andrewheiss.com/silent-skywalk/notebook/manuscript.html),
the journal doesn't consider the bibliography as part of the overall
word count (YAY), so I was able to include the appendix in the same
document and cross reference the figures and tables in the appendix from
the main document. It was *so nice* to not worry about two documents. I
wrote this:

::: {#cb4 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .default .code-with-copy}
We present the posterior distributions of the marginal means and AMCEs
for each of our experimental conditions in @fig-all-results and provide 
posterior medians, credible intervals, and other model diagnostics in 
@apptbl-all-results and @apptbl-coefs-orig.
```
:::

...and thanks to [Quarto's new ability to have custom cross reference
types](https://quarto.org/docs/authoring/cross-references-custom.html#example-supplemental-figures),
I got this when rendering:

> We present the posterior distributions of the marginal means and AMCEs
> for each of our experimental conditions in Figure 1 and provide
> posterior medians, credible intervals, and other model diagnostics in
> Table A5 and Table A8.

Holy grail achieved!

Except not quite.

[In another paper I wrapped up last
week](https://stats.andrewheiss.com/compassionate-clam/), the
bibliography counted as part of the total word count, and I had
citations in the appendix and they were getting added to the main
document's word count and inflating it. So I had to resort to my old
process of using two files: one for the manuscript and one for the
appendix. To cross reference stuff, I had to type the figure and table
names manually. Boooo.

Fortunately smarter people than me have developed
[`multibib`](https://github.com/pandoc-ext/multibib), a pandoc extension
that allows you to have multiple bibliographies and place them in
separate locations in the document. That means I can theoretically
create a main references list and a separate appendix references list,
and any references I have in the appendix won't get included in the main
list and won't add to the word count of the actual document.

Getting the extension to work with Quarto requires a little bit of extra
work, with some extra settings in the YAML metadata, but it works!

Holy grail *actually* achieved!
:::

::: {#how-to-get-quarto-to-work-with-multiple-bibliographies .section .level2 .page-columns .page-full}
## How to get Quarto to work with multiple bibliographies {#how-to-get-quarto-to-work-with-multiple-bibliographies .anchored anchor-id="how-to-get-quarto-to-work-with-multiple-bibliographies"}

Instead of explaining each step in detail, I'll just [link to a complete
example at GitHub](https://github.com/andrewheiss/multibib-testing). Go
clone or download the project there to see everything.

::: {.callout .callout-style-default .callout-tip .callout-titled}
::: {.callout-header .d-flex .align-content-center}
::: {.callout-icon-container}
:::

::: {.callout-title-container .flex-fill}
Complete example
:::
:::

::: {.callout-body-container .callout-body}
[Download the complete example from
GitHub.](https://github.com/andrewheiss/multibib-testing)
:::
:::

**IMPORTANT: This only works with Quarto 1.4**, which is currently a
pre-release. [You can download it from
GitHub](https://github.com/quarto-dev/quarto-cli/releases).

The example project has a couple Quarto extensions included in it
already, but if you're making a project from scratch and you want to
both use multiple bibliographies and count words, install the
[`multibib`](https://github.com/pandoc-ext/multibib) and
[`quarto-wordcount`](https://github.com/andrewheiss/quarto-wordcount)
extensions from your terminal:

::: {#cb5 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .bash .code-with-copy}
quarto add pandoc-ext/multibib
quarto add andrewheiss/quarto-wordcount
```
:::

Here's what the final rendered document looks like:

::: {.column-screen-inset-right .iframe-parent}
:::

::: {#bibliography-as-list .section .level3}
### Bibliography as list {#bibliography-as-list .anchored anchor-id="bibliography-as-list"}

Notice how the YAML uses this to define the separate bibliography files:

::: {#cb6 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .yaml .code-with-copy}
bibliography:
  main: references.json
  appendix: appendix.json
```
:::

This makes Quarto mad (see
[this](https://github.com/pandoc-ext/multibib/issues/9) and
[this](https://github.com/quarto-dev/quarto-cli/discussions/7642)):

::: {.img-fluid .quarto-figure .quarto-figure-center}
<figure class="figure">
<p><img
src="https://www.andrewheiss.com/blog/2023/12/11/separate-bibliographies-quarto/img/validation-error.png"
class="img-fluid figure-img" style="width:60.0%" /></p>
<figcaption>YAML validation error</figcaption>
</figure>
:::

This is because Quarto enforces a strict schema for all its YAML
settings and tries to make you input correct data for the different
settings. It also makes it so that tab completion of the different
settings works really well. It's a Good Thing. Technically, Quarto wants
the `bibliography` setting to be a string, or the name of a file with
the references. It doesn't want to work with a list of strings.

To get around this, we need to turn off the YAML validation. RStudio and
Visual Studio Code will still warn that the syntax is wrong, but the
document will render.

Alternatively, [there's a fork of
multibib](https://github.com/Mellich/pandoc-multibib) that uses separate
keys like this

::: {#cb7 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .yaml .code-with-copy}
bibliography_main: blah.bib
bibliography_appendix: blah.bib
```
:::
:::

::: {#fenced-divs-with-ids .section .level3}
### Fenced divs with IDs {#fenced-divs-with-ids .anchored anchor-id="fenced-divs-with-ids"}

To get the reference lists to appear where you want them in your
document, you have to include [fenced
divs](https://pandoc.org/MANUAL.html#divs-and-spans) with the id of each
of the bibliographies. Put them wherever you want in your document and
the corresponding reference list will go there.

::: {#cb8 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .default .code-with-copy}
Here's the main references list:

::: {#refs-main}
:::

Here's the appendix references list:

::: {#refs-appendix}
:::
```
:::
:::

::: {#disable-citeproc .section .level3}
### Disable citeproc {#disable-citeproc .anchored anchor-id="disable-citeproc"}

Pandoc uses citeproc to process citations and convert them to whatever
style you're using (Chicago, APA, MLA, etc). For the multiple
bibliography extension to work, pandoc has to see the different
bibliography entries before they're converted into text---the extension
does its magic sorting and dividing and *then* runs each of the
separated reference lists through citeproc. So we need to turn it off,
since multibib does it on its own.

::: {#cb9 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .yaml .code-with-copy}
citeproc: false
```
:::
:::

::: {#define-custom-cross-reference-types .section .level3}
### Define custom cross reference types {#define-custom-cross-reference-types .anchored anchor-id="define-custom-cross-reference-types"}

To get number prefixes like "Figure A2" and "Table A3", we need to
define [custom cross reference
types](https://quarto.org/docs/authoring/cross-references-custom.html#example-supplemental-figures):

::: {#cb10 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .yaml .code-with-copy}
crossref:
  custom:
    - kind: float
      key: apptbl
      latex-env: apptbl
      reference-prefix: Table A
      space-before-numbering: false
      latex-list-of-description: Appendix Table
```
:::

We can then use things like `@apptbl-whatever` and get automatic "A"
prefixes.
:::

::: {#counting-words .section .level3}
### Counting words {#counting-words .anchored anchor-id="counting-words"}

This magically works with [my word count
extension](https://github.com/andrewheiss/quarto-wordcount)! As long we
specify it as a filter *after* the multibib filter, everything will
work.

::: {#cb11 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .yaml .code-with-copy}
# Use the multibib and word count extensions
filters:
  - at: pre-render
    path: "_extensions/pandoc-ext/multibib/multibib.lua"
  - at: pre-render
    path: "_extensions/andrewheiss/wordcount/wordcount.lua"
```
:::

We'll get this output in the terminal:

::: {#cb12 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .default .code-with-copy}
83 in the main text + references, with 70 in the appendix
---------------------------------------------------------
26 words in text body
57 words in reference section
70 words in appendix section
```
:::

We only have to make one modification to the word count Lua filter
(`wordcount.lua`). Ordinarily, when rendering to HTML, the reference
list appears in a div with the id `refs`, so my extension looks for a
block with the `refs` ID:

::: {#cb13 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .lua .code-with-copy}
function is_ref_div (blk)
   return (blk.t == "Div" and blk.identifier == "refs")
end
```
:::

Since the main reference list now gets inserted into a div with the id
`refs-main`, the script needs to change:

::: {#cb14 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .lua .code-with-copy}
function is_ref_div (blk)
   return (blk.t == "Div" and blk.identifier == "refs-main")
end
```
:::

If I were better at Lua, I could probably figure out a way to make that
reference div setting more robust or maybe even configurable (like a
YAML key named `wordcount-refs-id` or something? idk). For now, I'm fine
just manually changing the Lua script as needed.
:::
:::

::: {#caveats-and-downsides .section .level2}
## Caveats and downsides {#caveats-and-downsides .anchored anchor-id="caveats-and-downsides"}

::: {#extra-warnings-i-cant-turn-off .section .level3}
### Extra warnings I can't turn off {#extra-warnings-i-cant-turn-off .anchored anchor-id="extra-warnings-i-cant-turn-off"}

One slightly annoying downside to this approach is that in the terminal
output, every citation is seen as missing:

::: {#cb15 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .default .code-with-copy}
[WARNING] Citeproc: citation Stan-2-26-1 not found
[WARNING] Citeproc: citation rproject-4-3-1 not found
[WARNING] Citeproc: citation Lovelace1842 not found
[WARNING] Citeproc: citation Turing1936 not found
```
:::

That's probably due the ordering of the filter (see
[this](https://github.com/quarto-dev/quarto-cli/discussions/7642#discussioncomment-7631536))---it's
getting run at some incorrect point in the whole Quarto rendering
process.

I don't know how to fix it though, so for now I'm just living with the
warnings :(

I'd love to figure this out someday though!
:::

::: {#manually-separating-the-pdf .section .level3}
### Manually separating the PDF {#manually-separating-the-pdf .anchored anchor-id="manually-separating-the-pdf"}

In the example Quarto file, I added a `\newpage` before the appendix
section so that the appendix would start on a new page. Most journal
submission sites require that you upload separate documents for the main
manuscript and the appendix.

To do this, I open the rendered PDF in Acrobat or macOS Preview and drag
the appendix section out manually.

I suppose I could automate it somehow with a command line tool like
[pdftk](https://www.pdflabs.com/tools/pdftk-the-pdf-toolkit/):

::: {#cb16 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .bash .code-with-copy}
# Extract pages 1-5
pdftk manuscript.pdf cat 1-5 output actual-paper.pdf

# Extract pages 6-9
pdftk manuscript.pdf cat 6-9 output appendix.pdf
```
:::

But programmatically figuring out those page ranges seems too hard and
not worth the effort---following [xkcd's automation
chart](https://xkcd.com/1205/), it won't save any time.
:::
:::

::: {#quarto-appendix .default}
::: {#quarto-citation .section .quarto-appendix-contents}
## Citation {#citation .anchored .quarto-appendix-heading}

<div>

::: {.quarto-appendix-secondary-label}
BibTeX citation:
:::

``` {.sourceCode .code-with-copy .quarto-appendix-bibtex}
@online{heiss2023,
  author = {Heiss, Andrew},
  title = {How to Create Separate Bibliographies in a {Quarto} Document},
  date = {2023-12-11},
  url = {https://www.andrewheiss.com/blog/2023/12/11/separate-bibliographies-quarto},
  doi = {10.59350/5dvez-q6817},
  langid = {en}
}
```

::: {.quarto-appendix-secondary-label}
For attribution, please cite this work as:
:::

::: {#ref-heiss2023 .csl-entry .quarto-appendix-citeas}
Heiss, Andrew. 2023. "How to Create Separate Bibliographies in a Quarto
Document." December 11, 2023. <https://doi.org/10.59350/5dvez-q6817>.
:::

</div>
:::
:::
