Charles Tapley Hoyt

The [Unified Medical Language System
(UMLS)](https://www.nlm.nih.gov/research/umls) is a widely used
biomedical and clinical vocabulary maintained by the United States
National Library of Medicine. However, it is notoriously difficult to
access and work with due to licensing restrictions and its complex
download system. In the same vein as my previous posts about
[DrugBank](/2020/12/14/taming-drugbank.html) and
[ChEMBL](/2021/08/05/taming-chembl-sql.html), this post describes [open
source software](https://github.com/cthoyt/umls_downloader) I've
developed for downloading and working with this data. It also works for
[RxNorm](https://www.nlm.nih.gov/research/umls/rxnorm/docs/rxnormfiles.html),
[SemMedDB](https://lhncbc.nlm.nih.gov/ii/tools/SemRep_SemMedDB_SKR/SemMedDB_download.html),
[SNOMED-CT](https://www.nlm.nih.gov/healthit/snomedct/international.html),
and any other data accessible through the [UMLS Terminology Services
(UTS) ticket granting
system](https://documentation.uts.nlm.nih.gov/automating-downloads.html).

The first big issue with the UMLS is its licensing. Here's an excerpt
from the [How to License and Access the Unified Medical Language System®
(UMLS®) Data](https://www.nlm.nih.gov/databases/umls.html) page accessed
on August 28^th^, 2023:

> 1.  Please sign up for a new UMLS Terminology Services (UTS) account
>     with your preferred identity provider at the UTS homepage.
> 2.  Complete and submit the license request form. NLM will send the
>     license approval e-mail within 5 business days after reviewing
>     your authenticated license request.
> 3.  You will sign in using identity provider credentials to download
>     files or access web interfaces that require UTS authentication
>     such as the UTS, VSAC, SNOMED CT, or RxNorm.

These are a few big hurdles:

1.  We typically expect scientific data to be available for download
    without login. Specifically, most data can be downloaded by
    following a link that points directly to a file. For example, ChEMBL
    v33 can be downloaded as a gzipped SQLite file from
    <https://ftp.ebi.ac.uk/pub/databases/chembl/ChEMBLdb/latest/chembl_33_sqlite.tar.gz>.
    Rather than providing a data download link, UMLS, has a complicated
    API called the UMLS Terminology Services (UTS) ticket granting
    system that needs to be asked for a specific file, polled for a
    unique access key, then downloaded via an ephemeral (i.e.,
    disappearing) URL that only works once.
2.  We typically expect scientific data to be licensed under a standard,
    widely used license such as those from [Creative
    Commons](https://creativecommons.org). Using well-understood
    licenses reduces the cognitive and legal burden of consumers when
    deciding if and how they can reuse, modify, or redistribute data.
    UMLS uses a non-standard [terms of
    service](https://www.nlm.nih.gov/databases/umls.html) that makes it
    more difficult to understand how the data can be stored, modified,
    or redistributed.
3.  Further, we hope that data is permissively licensed such that it can
    be re-used, modified, and re-distributed. The Creative Commons [CC
    BY 4.0](https://creativecommons.org/licenses/by/4.0/) and
    [CC0](https://creativecommons.org/publicdomain/zero/1.0/) licenses
    are golden standards of permissibility. UMLS does not have a
    permissive license, meaning (from my best interpretation) that you
    can not redistribute UMLS and you can (probably) not redistribute
    data derived from UMLS. As an aside, Creative Commons also has
    license containing clauses to be explicit about restrictions such as
    the share-alike (SA), non-commercial (NC), and non-distribution
    (ND). While these clauses aren't ideal for scientific data, it would
    at least be nice for UMLS to use a Creative Commons license with the
    appropriate combination of these clauses (I guess all three) to make
    it more explicit about its restrictions.
4.  The most bizarre facet of UMLS is that they require you to fill out
    a user survey each year to keep access.

I want to 1) convert UMLS into an [OWL](https://www.w3.org/OWL/)
ontology and 2) extract and encode its semantic mappings to external
vocabularies like the [Medical Subject Headings
(MeSH)](https://bioregistry.iomesh) with [Simple Standard for Sharing
Ontology Mappings (SSSOM)](https://github.com/mapping-commons/sssom).
Given all of these hurdles, it's probably the case that I am not allowed
to redistribute these artifacts.

All together, I consider this a big bummer. The United States National
Library of Medicine (NLM) maintains several highly influential
resources, but I have found in many instances that they lack a community
perspective. Regardless, even as an expat, I pay American taxes, and it
makes me upset that the government funds the development and maintenance
of resources that I can't easily use.

## How To Break Free

![A clip of Queen\'s
](https://media.tenor.com/lfjcIHfOhqoAAAAC/freddie-mercury-queen.gif){i=""
want="" to="" break="" free\"="" music="" video\"=""}

Despite all of this rigamarole, there's a process to subvert these
issues by automating the interaction with the [UMLS Terminology Services
(UTS)](https://uts.nlm.nih.gov/uts) and therefore enabling automated
download of UMLS and the following (non-exhaustive) list of resources:

- [RxNorm](https://www.nlm.nih.gov/research/umls/rxnorm/docs/rxnormfiles.html)
- [SemMedDB](https://lhncbc.nlm.nih.gov/ii/tools/SemRep_SemMedDB_SKR/SemMedDB_download.html)
- [SNOMED-CT](https://www.nlm.nih.gov/healthit/snomedct/international.html)
- potentially more in the future

This has been implemented in the open source
[`umls_downloader`{.language-plaintext
.highlighter-rouge}](https://github.com/cthoyt/umls-downloader) Python
package. It can be installed with the following one-liner in your shell:

:::: {.language-shell .highlighter-rouge}
::: {.highlight}
``` highlight
$ pip install umls_downloader
```
:::
::::

Below, I'll walk you through using it. Throughout, keep in mind that
full documentation for the `umls_downloader`{.language-plaintext
.highlighter-rouge} is available at
[umls-downloader.readthedocs.io](https://umls-downloader.readthedocs.io),
which describes the other functionality and other data that can be
downloaded.

### Usage

UMLS has three different distributions that are described
[here](https://www.nlm.nih.gov/research/umls/licensedcontent/umlsknowledgesources.html).
The following Python code downloads the most simple and straightforward
file, `MRCONSO.RRF`{.language-plaintext .highlighter-rouge} as a zip
archive:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from umls_downloader import download_umls

path = download_umls(version="2023AA", api_key="")
```
:::
::::

This code is smart and does not need to download the file more than
once. It uses [`pystow`{.language-plaintext
.highlighter-rouge}](https://github.com/cthoyt/pystow) to choose a
stable path `~/.data/bio/umls`{.language-plaintext .highlighter-rouge}
relative to the current user's home directory. Inside this directory, it
also uses the version of the data to create a subdirectory. Finally,
this function returns the path to the data, such that no file paths ever
need to be hard-coded.

> **Warning** This still requires an API key, which requires creating an
> account, agreeing to UMLS's terms and conditions, etc. This can be
> done here: <https://uts.nlm.nih.gov/uts/edit-profile>.

### Automating Configuration of UTS Credentials

There are two ways to automatically set the API key, so you don't have
to worry about getting it and passing it around in your python code:

1.  Set `UMLS_API_KEY`{.language-plaintext .highlighter-rouge} in the
    environment. This can be done in your interactive session or in the
    configuration for your shell such as in a
    `.bashrc`{.language-plaintext .highlighter-rouge} file for the
    Bourne Again Shell (bash).
2.  Create `~/.config/umls.ini`{.language-plaintext .highlighter-rouge}
    and set in the `[umls]`{.language-plaintext .highlighter-rouge}
    section a `api_key`{.language-plaintext .highlighter-rouge} key.
    Mine looks like:

    :::: {.language-ini .highlighter-rouge}
    ::: {.highlight}
    ``` highlight
    [umls]
    api_key=1234567890abcdefghijklmno
    ```
    :::
    ::::

Now you can omit the `api_key`{.language-plaintext .highlighter-rouge}
keyword like in the following:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from umls_downloader import download_umls

# Same path as before
path = download_umls(version="2023AA")
```
:::
::::

### Download the Latest Version

First, you'll have to install [`bioversions`{.language-plaintext
.highlighter-rouge}](https://github.com/cthoyt/bioversions) with
`pip install bioversions`{.language-plaintext .highlighter-rouge}, whose
job it is to look up the latest version of many databases. Then, you can
modify the previous code slightly by omitting the
`version`{.language-plaintext .highlighter-rouge} keyword argument:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from umls_downloader import download_umls

# Same path as before (when run on September 1st, 2023)
path = download_umls()
```
:::
::::

### Download and open the file

The UMLS file is zipped, so it's usually accompanied by the following
boilerplate code:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
import zipfile
from umls_downloader import download_umls

path = download_umls()
with zipfile.ZipFile(path) as zip_file:
    with zip_file.open("MRCONSO.RRF", mode="r") as file:
        for line in file:
            ...
```
:::
::::

This exact code is wrapped with the
`umls_downloader.open_umls()`{.language-plaintext .highlighter-rouge}
using Python's context manager, so it can more simply be written as:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from umls_downloader import open_umls

with open_umls() as file:
    for line in file:
        ...
```
:::
::::

> **Note** The `version`{.language-plaintext .highlighter-rouge} and
> `api_key`{.language-plaintext .highlighter-rouge} arguments work the
> same for `umls_downloader.open_umls()`{.language-plaintext
> .highlighter-rouge} as in
> `umls_downloader.download_umls()`{.language-plaintext
> .highlighter-rouge}

At this point, it's up to you to decide how you want to consume the
`MRCONSO.RRF`{.language-plaintext .highlighter-rouge} file. Below, I
give a demo on how parsed this file in PyOBO in order to convert UMLS to
an OWL ontology.

## Why not an API?

The UMLS provides an
[API](https://documentation.uts.nlm.nih.gov/rest/home.html) for access
to tiny bits of data at a time. There are even two recent (last 5 years)
packages [`umls-api`{.language-plaintext
.highlighter-rouge}](https://pypi.org/project/umls-api)
[`connect-umls`{.language-plaintext
.highlighter-rouge}](https://pypi.org/project/connect-umls) that provide
a wrapper around them. However, API access is generally rate limited,
difficult to use in bulk, and slow. For working with UMLS (or any other
database, for that matter) in bulk, it's necessary to download full
database dumps.

## UMLS Conversions

Building on top of the automated download of UMLS, I implemented a
fit-for-purpose processor with the [PyOBO
framework](https://github.com/pyobo/pyobo) that converts UMLS into an
ontology (encoded either as OWL, OBO, or OBO Graph JSON) which can
therefore be used to generate semantic mappings in the SSSOM format. The
code that implements this can be found
[here](https://github.com/pyobo/pyobo/commit/be67eadf77afa270cf224cad71d5b1905ae78b53).
After installing PyOBO with `pip install pyobo`{.language-plaintext
.highlighter-rouge}, you can automatically download and convert UMLS
first into an ontology encoded in the [OBO flat file
format](https://owlcollab.github.io/oboformat/doc/GO.format.obo-1_4.html),
then convert to OWL with the following code. Note: you'll need
[`robot`{.language-plaintext
.highlighter-rouge}](https://robot.obolibrary.org/) for the second step:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
import pyobo

umls = pyobo.get_ontology("umls")

# Write simple OBO Format
umls.write_obo("umls.obo")

# Convert to OWL
from pyobo.utils.misc import obo_to_owl

obo_to_owl("umls.obo", "umls.owl")
```
:::
::::

In an ideal world, the results of such a conversion could be included as
a part of the [OBO Database
Ingestion](https://github.com/biopragmatics/obo-db-ingest), which
converts database resources available through PyOBO into ontology
artifacts, archives them on GitHub and Zenodo, and gives them PURLs all
on a weekly basis to make sure the most up-to-date version is available
as well as all previous named versions. Instead, we live in a world with
pineapple pizza and restrictive licenses.

One of the nice qualities of UMLS is that it is a semantic mapping hub.
It provides mostly complete mappings between many vocabularies including
[MeSH](https://bioregistry.io/mesh),
[NCIT](https://bioregistry.io/ncit),
[SNOMED-CT](https://bioregistry.io/snomedct),
[HPO](https://bioregistry.io/hpo),
[LOINC](https://bioregistry.io/loinc), and more. However, there are a
few caveats to consider:

1.  UMLS mappings aren't all 1-to-1. For example, MeSH mappings
    typically include many UMLS terms (narrower) pointing to the same
    MeSH term (broader). For other vocabularies, such as NCBITaxon, UMLS
    mappings are more reliably 1-to-1. Thanks to Tiago Lubiana for
    pointing this out.
2.  Mapping provenance is not available, so the
    `mapping_justification`{.language-plaintext .highlighter-rouge}
    field in SSSOM is uniformly filled with
    `sempav:UnspecifiedMatching`{.language-plaintext
    .highlighter-rouge}.
3.  Similarly, UMLS does not apply precise semantic predicates for each
    mapping. This means that they are output in PyOBO and as SSSOM with
    `oboInOwl:hasDbXref`{.language-plaintext .highlighter-rouge} instead
    of more detailed types such as `skos:exactMatch`{.language-plaintext
    .highlighter-rouge}, `skos:narrowMatch`{.language-plaintext
    .highlighter-rouge}, and `skos:broaderMatch`{.language-plaintext
    .highlighter-rouge}. Tools like
    [Boomer](https://github.com/INCATools/boomer) can be used to address
    this (in part). The [Semantic Mapping Reasoning Assembler
    (SeMRA)](https://github.com/biopragmatics/semra) can also be
    configured with prior knowledge about UMLS mapping assumptions when
    aggregating and reasoning over semantic mappings at scale.

With that in mind, anything that can be loaded as an ontology in PyOBO
can also be exported with SSSOM, which I show below. For UMLS, this
looks like:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
import pyobo

df = pyobo.get_sssom_df("umls", names=False)
df.to_csv("umls.sssom.tsv", sep="\t", index=False)
```
:::
::::

> **Note** You can set `names=True`{.language-plaintext
> .highlighter-rouge} to have PyOBO look up the names for all entities,
> but this is a bit of a rabbit hole since it requires getting and
> processing many external resources.

------------------------------------------------------------------------

There's much more to say about UMLS and SSSOM, but this is a good place
to pause and publish this post, since getting UMLS as SSSOM is a task a
lot of people have asked me for help with lately. I might also come back
and explain more about how I use the other resources from UMLS's UTS.


The Unified Medical Language System (UMLS) is a widely used biomedical and clinical vocabulary maintained by the United States National Library of Medicine. However, it is notoriously difficult to access and work with due to licensing restrictions and its complex download system. In the same vein as my previous posts about DrugBank and ChEMBL, this post describes open source software I’ve developed for downloading and working with this data.

Messages de Rogue Scholar

Unlocking UMLS