Unraveling complex biology with biological knowledge graphs. Content licensed under CC BY 4.0.

Charles Tapley Hoyt

I’ve just returned from the 17

 th

Annual International Biocuration Conference at the Indian Biological Data Centre (IBDC) in Faridabad, India.

Discussions and Follow-ups from Biocuration 2024

Using Pydantic for encoding data models and FastAPI for implementing APIs on top of them has become a staple for many Python programmers.

Semantic Pydantic

I finally got back into reading! Over winter break 2022, I started the Stormlight Archive then followed up in 2023 by reading the entirety of Brandon Sanderson’s

 Cosmere

, as well as a some other fantasy, science fiction, and literary fiction. Here’s the list.

Books I Read in 2023

The Unified Medical Language System (UMLS) is a widely used biomedical and clinical vocabulary maintained by the United States National Library of Medicine. However, it is notoriously difficult to access and work with due to licensing restrictions and its complex download system. In the same vein as my previous posts about DrugBank and ChEMBL, this post describes open source software I’ve developed for downloading and working with this data.

Unlocking UMLS

Today’s short post is about three SPARQL queries I wrote to get bibliometric information about journals and publishers out of Wikidata. Each of the following queries can be readily copy-pasted into the Wikidata Query Service and run in the browser.

Querying Journals and Publishers in Wikidata

I was recently nominated for the International Society for Biocuration’s Excellence in Biocuration Early Career Award (results will be announced on June 14

 th

!). This made me curious about how to model nominations and awards on Wikidata. In this post, I’ll describe how to curate awards, nominations, recipients, and how to make SPARQL queries to get them.

Modeling and Querying Awards in Wikidata

Archival Resource Keys (ARKs) are flavor of persistent identifiers like DOIs, URNs, and Handles that have the benefit of being free, flexible with what metadata gets attached, and natively able to resolve to web pages. Name-to-Thing (N2T) implements a resolver for a variety of ARKs, so this blog post is about how that resolver can be re-implemented with the curies Python package. In a lot of ways, ARKs look and act like CURIEs.

Re-implementing the N2T ARK Resolver

I’ve been working on improving reproducibility in the field of cheminformatics for some time now. For example, I’ve written posts about making data from DrugBank and ChEMBL more actionable. Over the last year, I’ve been preparing a concept with the editors of the Journal of Cheminformatics on how to include an assessment of reproducibility to reviews of manuscripts submitted to the journal.

Reproducibility Pilot in the Journal of Cheminformatics

Several controlled vocabularies and ontologies that aren’t themselves OBO Foundry ontologies use unsanctioned OBO PURLs. This post is about how to use the Bioregistry to identify which resources are doing this and to give some insight into how we arrived in this situation. Background on the OBO Foundry   The OBO Foundry is a set of independent, interoperable biomedical ontologies that aspire to using shared development principles.

Resources masquerading as OBO Foundry ontologies

This blog is normally about

 very serious science

, but I’m taking a break from that for the evening to advertize my band’s upcoming show on April 8

 th

in the SPH Music Masters Finale (aka, the German Battle of the Bands). We need your support!

The Representatives of Monkey Jack - German Battle of the Bands Finale

The Open Researcher and Contributor Identifier (ORCID) database is an invaluable resource that supports the unambiguous identification of researchers. However, its first party data dump is too complex, verbose, and unstandardized for many use cases. This post describes open source software I wrote that automates downloading, processing, and exporting ORCID into a more usable form. I put the results on Zenodo under the CC0 license.

Easier ORCID

Biopragmatics

Using [Pydantic](https://docs.pydantic.dev) for encoding data models and
[FastAPI](https://fastapi.tiangolo.com) for implementing APIs on top of
them has become a staple for many Python programmers. When this
intersects with the semantic web, linked open data, and the natural
sciences, we are still lacking a bridge to annotate our data models and
APIs to make them more FAIR (findable, accessible, interoperable, and
reusable). In this post, we build an extension to Pydantic and FastAPI
to annotate data models' fields and API endpoints' query, path, and
other parameters using the [Bioregistry](https://bioregistry.io), a
comprehensive catalog of metadata about semantic spaces from the
semantic web and the natural sciences.

As a demonstration, we will build a data model and API that serves
information about scholars.

## First Steps with Pydantic

We'll use [Open Researcher and Contributor (ORCID)](https://orcid.org/)
identifiers as primary keys, include the researcher's name, and start
with a single cross-reference, e.g., to the author's
[DBLP](https://dblp.org/) identifier. We'll encode this data model using
[Pydantic](https://docs.pydantic.dev/latest/) in the Python programming
language as follows:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from pydantic import BaseModel, Field


class ScholarV1(BaseModel):
    """A model representing a researcher, who might have several IDs on different services."""

    orcid: str = Field(...)
    name: str = Field(...)
    dblp: str | None = Field(None)


print(ScholarV1.schema_json(indent=2))
```
:::
::::

JSON Schema - Version 1

\`\`\`json { \"title\": \"ScholarV1\", \"description\": \"A model
representing a researcher, who might have several IDs on different
services.\", \"type\": \"object\", \"properties\": { \"orcid\": {
\"title\": \"Orcid\", \"type\": \"string\" }, \"name\": { \"title\":
\"Name\", \"type\": \"string\" }, \"dblp\": { \"title\": \"Dblp\",
\"type\": \"string\" } }, \"required\": \[ \"orcid\", \"name\" \] }
\`\`\`

There are several places for improvement here:

1.  Correct capitalization of the titles (`ORCID`{.language-plaintext
    .highlighter-rouge} instead of `Orcid`{.language-plaintext
    .highlighter-rouge} and `DBLP`{.language-plaintext
    .highlighter-rouge} instead of `Dblp`{.language-plaintext
    .highlighter-rouge})
2.  Add useful descriptions of what each field is
3.  Have regular expression patterns to validate input
4.  Give an example

All of these are possible to annotate into Pydantic's
`Field`{.language-plaintext .highlighter-rouge} object, but it requires
lots of effort and takes lots of space. Even worse, this might have to
be partially duplicated if multiple models share the same fields. In the
example below, I annotated ORCID but will skip the others for brevity.

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from pydantic import BaseModel, Field


class ScholarV2(BaseModel):
    """A model representing a researcher, who might have several IDs on different services."""

    orcid: str = Field(
        ...,
        title="ORCID",
        description="A stable, public identifier for a researcher from https://orcid.com",
        pattern="^\d{4}-\d{4}-\d{4}-\d{3}(\d|X)$",
        example="0000-0003-4423-4370",
    )
    name: str = Field(...)
    dblp: str | None = Field(None)


print(ScholarV2.schema_json(indent=2))
```
:::
::::

JSON Schema - Version 2

\`\`\`json { \"title\": \"ScholarV2\", \"description\": \"A model
representing a researcher, who might have several IDs on different
services.\", \"type\": \"object\", \"properties\": { \"orcid\": {
\"title\": \"ORCID\", \"description\": \"A stable, public identifier for
a researcher from https://orcid.com\", \"pattern\":
\"\^\\d{4}-\\d{4}-\\d{4}-\\d{3}(\\d\|X)\$\", \"example\":
\"0000-0003-4423-4370\", \"type\": \"string\" }, \"name\": { \"title\":
\"Name\", \"type\": \"string\" }, \"dblp\": { \"title\": \"Dblp\",
\"type\": \"string\" } }, \"required\": \[ \"orcid\", \"name\" \] }
\`\`\`

However, this was a lot of work. It would be nice if there were some
database of all the semantic spaces in the semantic web and natural
sciences that contained the name, description, regular expression
pattern, and examples. Then, we could draw from this database to
automatically populate our fields.

## Adding Some Bioregistry Magic

The good news is that such a database exists - it's called the
[Bioregistry](https://bioregistry.io). Each semantic space (e.g., ORCID,
DBLP) gets a *prefix* which is usually an acronym for the name of the
resource that serves as the primary key for the semantic space. These
prefixes are also useful in making references to entities in the
semantic space more FAIR (findable, accessible, interoperable, reusable)
using the [compact URI (CURIE)
syntax](https://cthoyt.com/2021/09/14/curies.html), though this isn't
the goal of this demo.

I've mocked some Python code that bridges Pydantic and the Bioregistry
in this repository (https://github.com/cthoyt/semantic-pydantic). I'm
calling it **Semantic Pydantic** because it lets us annotate our data
models with external metadata (and because it rhymes).

Here's the same model as before, but now using a
`SemanticField`{.language-plaintext .highlighter-rouge} that extends
Pydantic's `Field`{.language-plaintext .highlighter-rouge}. It has a
special keyword `prefix`{.language-plaintext .highlighter-rouge} that
lets you give a Bioregistry prefix, then it is smart enough to fill out
all the fields on its own. I also took the liberty of adding several
more semantic spaces that identify scholars like [Web of Science
(`wos`{.language-plaintext
.highlighter-rouge})](https://bioregistry.io/wos.researcher),
[Scopus](https://bioregistry.io/scopus), and even
[GitHub](https://bioregistry.io/github).

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from pydantic import BaseModel, Field

from semantic_pydantic import SemanticField


class ScholarV3(BaseModel):
    """A model representing a researcher, who might have several IDs on different services."""

    orcid: str = SemanticField(..., prefix="orcid")
    name: str = Field(..., example="Charles Tapley Hoyt")

    wos: str | None = SemanticField(default=None, prefix="wos.researcher")
    dblp: str | None = SemanticField(default=None, prefix="dblp.author")
    github: str | None = SemanticField(default=None, prefix="github")
    scopus: str | None = SemanticField(default=None, prefix="scopus")
    semion: str | None = SemanticField(default=None, prefix="semion")
    publons: str | None = SemanticField(default=None, prefix="publons.researcher")
    authorea: str | None = SemanticField(default=None, prefix="authorea.author")


print(ScholarV3.schema_json(indent=2))
```
:::
::::

Finally, we can see a very detailed JSON schema, which includes
everything from before plus additional context from the Bioregistry,
including the prefix itself as well as mappings from the Bioregistry
prefix to external registries like
[BARTOC](https://bioregistry.io/metaregistry/bartoc),
[FAIRsharing](https://bioregistry.io/metaregistry/fairsharing), and
others. Together, these make the data model more FAIR and support
interoperability, since now it is possible to directly match the fields
annotated with Bioregistry prefixes in this model to fields annotated
with the same prefix in other models, even external to the project.

JSON Schema - Version 3

\`\`\`json { \"title\": \"ScholarV3\", \"description\": \"A model
representing a researcher, who might have several IDs on different
services.\", \"type\": \"object\", \"properties\": { \"orcid\": {
\"title\": \"Open Researcher and Contributor\", \"description\": \"

This field corresponds to a local unique identifier from *Open
Researcher and Contributor*.\n

#### Provenance

The semantics of this field are derived from
the\n[`orcid`](\%22https://bioregistry.io/orcid\%22) entry in\nthe
[Bioregistry](\%22https://bioregistry.io\%22): a registry of semantic
web and linked \nopen data compact URI (CURIE) prefixes and URI
prefixes.\n

#### Description of Semantic Space

ORCID (Open Researcher and Contributor ID) is an open, non-profit,
community-based effort to create and maintain a registry of unique
identifiers for individual researchers. ORCID records hold non-sensitive
information such as name, email, organization name, and research
activities.\", \"pattern\": \"\^\\d{4}-\\d{4}-\\d{4}-\\d{3}(\\d\|X)\$\",
\"example\": \"0000-0003-4423-4370\", \"json_schema_extra\": {
\"bioregistry\": { \"prefix\": \"orcid\", \"mappings\": { \"bartoc\":
\"2021\", \"biocontext\": \"ORCID\", \"biolink\": \"ORCID\",
\"fairsharing\": \"FAIRsharing.nx58jg\", \"go\": \"orcid\", \"miriam\":
\"orcid\", \"n2t\": \"orcid\", \"wikidata\": \"P496\" } } }, \"type\":
\"string\" }, \"name\": { \"title\": \"Name\", \"example\": \"Charles
Tapley Hoyt\", \"type\": \"string\" }, \... } } \`\`\` Note that the
extra semantic fields have been truncated for brevity. Each of them also
has lots of detailed information.

## Web Application

Let's take the next step to a web application using FastAPI. The goal of
this web application will be to look up the information for a scholar in
Wikidata based on their ORCID. You don't really have to understand how
the query works other than that it takes in an ORCID string and gives
back an instance of the Scholar model we've been working on above.

The app uses annotations for the query parameters, path parameters, and
other inputs to routes using extensions of Pydantic
`Fields`{.language-plaintext .highlighter-rouge}. So similar to before,
we can extend their custom fields to be semantic in **Semantic
Pydantic**.

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from fastapi import FastAPI
from semantic_pydantic import SemanticPath

app = FastAPI(title="Semantic Pydantic Demo")
Scholar = ...  # defined before


@app.get("/api/orcid/{orcid}", response_model=Scholar)
def get_scholar_from_orcid(orcid: str = SemanticPath(prefix="orcid")):
    """Get xrefs for a researcher in Wikidata, given ORCID identifier."""
    ...  # full implementation in https://github.com/cthoyt/semantic-pydantic
    return Scholar(...)
```
:::
::::

The real power is how this translates to the API, and more importantly,
the automatically generated API documentation. First, the
`SemanticPath`{.language-plaintext .highlighter-rouge} object which we
used in place of a normal `fastapi.Path`{.language-plaintext
.highlighter-rouge} also knows it is for ORCID identifiers. Second, the
response model points to the Scholar class from before which already
knows about its semantics. Below, we see this in a screenshot of the
OpenAPI (formerly known as Swagger) user interface automatically
generated by FastAPI.

![A screenshot of the Swagger UI, enriched by the semantic annotations
described above.](/img/semantic-pydantic-api.png)

There are two big things to note here:

1.  All the detailed information about ORCID makes it into the
    parameter, also giving an example for users to get started
2.  All the detailed information from the response model (Scholar) gets
    shown as a real example.

Now, we have an API that is also annotated with detailed semantics. If
you take a look at the OpenAPI JSON file, it has similar references to
Bioregistry prefixes for the routes themselves, and directly reuses the
JSON schema for the response model.

## Next Steps

So far, this is a proof-of-concept that lives in an *ad hoc* repository.
It's not clear yet if this code is just a neat demo, whether it should
live inside the Bioregistry Python package, I haven't decided yet if
this should go inside the [Bioregistry Python
package](https://github.com/biopragmatics/bioregistry/), or if it should
be in a stand-alone package that might be extensible even further. There
are a few other things to think about in the meantime:

### Infrastructure for FAIR Models and APIs

The first version of this idea just throws the Bioregistry data into the
JSON schema. It would be interesting to develop this infrastructure
further, such as keeping a catalog of all APIs that consume or produce
data models containing semantic fields. A few places this would be
great:

1.  The [Cheminformatics
    Microservice](https://github.com/Steinbeck-Lab/cheminformatics-microservice/issues)
    contains tons of references to [InChi Keys
    (`inchikey`{.language-plaintext
    .highlighter-rouge})](https://bioregistry.io/registry/inchikey)
2.  The Bioregistry could dog-food its own API
3.  The [INDRA Discovery API](https://discovery.indra.bio/apidocs) could
    be refactored to use semantic inputs and outputs

There are also so many more examples, please let me know some services
you think would benefit in the comments on my blog post. Looking
forward, it's also a question on how to automatically discover such
semantic APIs (e.g., by cleverly searching GitHub) or if it would have
to be a manually curated catalog.

### Resolving URLs

A key feature of the Bioregistry is that it provides a way to take a
local unique identifier for an entity in a given semantic space and make
a URL that points to a web page describing the entity. For example, if
you have an ORCID identifier, you can make a URL for the ORCID page
following the format `https://orcid.org/`{.language-plaintext
.highlighter-rouge}. It would be very cool to extend Semantic Pydantic
to add some properties that auto-generate URLs, like in the following:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from pydantic import BaseModel, Field

from semantic_pydantic import SemanticField


class Scholar(BaseModel):
    orcid: str = SemanticField(..., prefix="orcid")
    name: str = Field(...)


charlie = Scholar(orcid="0000-0003-4423-4370", name="Charles Tapley Hoyt")
assert charlie.orcid_url == 'https://orcid.org/0000-0003-4423-4370'
```
:::
::::

### Run the Demo

The demo can be run by cloning the repository, installing its
requirements, and running the self-contained
`app.py`{.language-plaintext .highlighter-rouge}.

:::: {.language-shell .highlighter-rouge}
::: {.highlight}
``` highlight
git clone https://github.com/cthoyt/semantic-pydantic
cd semantic-pydantic
python -m pip install -r requirements.txt
python app.py
```
:::
::::

------------------------------------------------------------------------

This is my first science post of 2024! I'm very happy that the
Bioregistry is currently supported by the Chan Zuckerberg Initiative
(CZI) under award 2023-329850.
