Unraveling complex biology with biological knowledge graphs. Content licensed under CC BY 4.0.

Charles Tapley Hoyt

I’ve just returned from the 17

 th

Annual International Biocuration Conference at the Indian Biological Data Centre (IBDC) in Faridabad, India.

Discussions and Follow-ups from Biocuration 2024

Using Pydantic for encoding data models and FastAPI for implementing APIs on top of them has become a staple for many Python programmers.

Semantic Pydantic

I finally got back into reading! Over winter break 2022, I started the Stormlight Archive then followed up in 2023 by reading the entirety of Brandon Sanderson’s

 Cosmere

, as well as a some other fantasy, science fiction, and literary fiction. Here’s the list.

Books I Read in 2023

The Unified Medical Language System (UMLS) is a widely used biomedical and clinical vocabulary maintained by the United States National Library of Medicine. However, it is notoriously difficult to access and work with due to licensing restrictions and its complex download system. In the same vein as my previous posts about DrugBank and ChEMBL, this post describes open source software I’ve developed for downloading and working with this data.

Unlocking UMLS

Today’s short post is about three SPARQL queries I wrote to get bibliometric information about journals and publishers out of Wikidata. Each of the following queries can be readily copy-pasted into the Wikidata Query Service and run in the browser.

Querying Journals and Publishers in Wikidata

I was recently nominated for the International Society for Biocuration’s Excellence in Biocuration Early Career Award (results will be announced on June 14

 th

!). This made me curious about how to model nominations and awards on Wikidata. In this post, I’ll describe how to curate awards, nominations, recipients, and how to make SPARQL queries to get them.

Modeling and Querying Awards in Wikidata

Archival Resource Keys (ARKs) are flavor of persistent identifiers like DOIs, URNs, and Handles that have the benefit of being free, flexible with what metadata gets attached, and natively able to resolve to web pages. Name-to-Thing (N2T) implements a resolver for a variety of ARKs, so this blog post is about how that resolver can be re-implemented with the curies Python package. In a lot of ways, ARKs look and act like CURIEs.

Re-implementing the N2T ARK Resolver

I’ve been working on improving reproducibility in the field of cheminformatics for some time now. For example, I’ve written posts about making data from DrugBank and ChEMBL more actionable. Over the last year, I’ve been preparing a concept with the editors of the Journal of Cheminformatics on how to include an assessment of reproducibility to reviews of manuscripts submitted to the journal.

Reproducibility Pilot in the Journal of Cheminformatics

Several controlled vocabularies and ontologies that aren’t themselves OBO Foundry ontologies use unsanctioned OBO PURLs. This post is about how to use the Bioregistry to identify which resources are doing this and to give some insight into how we arrived in this situation. Background on the OBO Foundry   The OBO Foundry is a set of independent, interoperable biomedical ontologies that aspire to using shared development principles.

Resources masquerading as OBO Foundry ontologies

This blog is normally about

 very serious science

, but I’m taking a break from that for the evening to advertize my band’s upcoming show on April 8

 th

in the SPH Music Masters Finale (aka, the German Battle of the Bands). We need your support!

The Representatives of Monkey Jack - German Battle of the Bands Finale

The Open Researcher and Contributor Identifier (ORCID) database is an invaluable resource that supports the unambiguous identification of researchers. However, its first party data dump is too complex, verbose, and unstandardized for many use cases. This post describes open source software I wrote that automates downloading, processing, and exporting ORCID into a more usable form. I put the results on Zenodo under the CC0 license.

Easier ORCID

Biopragmatics

The [Unified Medical Language System
(UMLS)](https://www.nlm.nih.gov/research/umls) is a widely used
biomedical and clinical vocabulary maintained by the United States
National Library of Medicine. However, it is notoriously difficult to
access and work with due to licensing restrictions and its complex
download system. In the same vein as my previous posts about
[DrugBank](/2020/12/14/taming-drugbank.html) and
[ChEMBL](/2021/08/05/taming-chembl-sql.html), this post describes [open
source software](https://github.com/cthoyt/umls_downloader) I've
developed for downloading and working with this data. It also works for
[RxNorm](https://www.nlm.nih.gov/research/umls/rxnorm/docs/rxnormfiles.html),
[SemMedDB](https://lhncbc.nlm.nih.gov/ii/tools/SemRep_SemMedDB_SKR/SemMedDB_download.html),
[SNOMED-CT](https://www.nlm.nih.gov/healthit/snomedct/international.html),
and any other data accessible through the [UMLS Terminology Services
(UTS) ticket granting
system](https://documentation.uts.nlm.nih.gov/automating-downloads.html).

The first big issue with the UMLS is its licensing. Here's an excerpt
from the [How to License and Access the Unified Medical Language System®
(UMLS®) Data](https://www.nlm.nih.gov/databases/umls.html) page accessed
on August 28^th^, 2023:

> 1. Please sign up for a new UMLS Terminology Services (UTS) account
> with your preferred identity provider at the UTS homepage.
> 2. Complete and submit the license request form. NLM will send the
> license approval e-mail within 5 business days after reviewing
> your authenticated license request.
> 3. You will sign in using identity provider credentials to download
> files or access web interfaces that require UTS authentication
> such as the UTS, VSAC, SNOMED CT, or RxNorm.

These are a few big hurdles:

1. We typically expect scientific data to be available for download
 without login. Specifically, most data can be downloaded by
 following a link that points directly to a file. For example, ChEMBL
 v33 can be downloaded as a gzipped SQLite file from
 <https://ftp.ebi.ac.uk/pub/databases/chembl/ChEMBLdb/latest/chembl_33_sqlite.tar.gz>.
 Rather than providing a data download link, UMLS, has a complicated
 API called the UMLS Terminology Services (UTS) ticket granting
 system that needs to be asked for a specific file, polled for a
 unique access key, then downloaded via an ephemeral (i.e.,
 disappearing) URL that only works once.
2. We typically expect scientific data to be licensed under a standard,
 widely used license such as those from [Creative
 Commons](https://creativecommons.org). Using well-understood
 licenses reduces the cognitive and legal burden of consumers when
 deciding if and how they can reuse, modify, or redistribute data.
 UMLS uses a non-standard [terms of
 service](https://www.nlm.nih.gov/databases/umls.html) that makes it
 more difficult to understand how the data can be stored, modified,
 or redistributed.
3. Further, we hope that data is permissively licensed such that it can
 be re-used, modified, and re-distributed. The Creative Commons [CC
 BY 4.0](https://creativecommons.org/licenses/by/4.0/) and
 [CC0](https://creativecommons.org/publicdomain/zero/1.0/) licenses
 are golden standards of permissibility. UMLS does not have a
 permissive license, meaning (from my best interpretation) that you
 can not redistribute UMLS and you can (probably) not redistribute
 data derived from UMLS. As an aside, Creative Commons also has
 license containing clauses to be explicit about restrictions such as
 the share-alike (SA), non-commercial (NC), and non-distribution
 (ND). While these clauses aren't ideal for scientific data, it would
 at least be nice for UMLS to use a Creative Commons license with the
 appropriate combination of these clauses (I guess all three) to make
 it more explicit about its restrictions.
4. The most bizarre facet of UMLS is that they require you to fill out
 a user survey each year to keep access.

I want to 1) convert UMLS into an [OWL](https://www.w3.org/OWL/)
ontology and 2) extract and encode its semantic mappings to external
vocabularies like the [Medical Subject Headings
(MeSH)](https://bioregistry.iomesh) with [Simple Standard for Sharing
Ontology Mappings (SSSOM)](https://github.com/mapping-commons/sssom).
Given all of these hurdles, it's probably the case that I am not allowed
to redistribute these artifacts.

All together, I consider this a big bummer. The United States National
Library of Medicine (NLM) maintains several highly influential
resources, but I have found in many instances that they lack a community
perspective. Regardless, even as an expat, I pay American taxes, and it
makes me upset that the government funds the development and maintenance
of resources that I can't easily use.

## How To Break Free

![A clip of Queen\'s
](https://media.tenor.com/lfjcIHfOhqoAAAAC/freddie-mercury-queen.gif){i=""
want="" to="" break="" free\"="" music="" video\"=""}

Despite all of this rigamarole, there's a process to subvert these
issues by automating the interaction with the [UMLS Terminology Services
(UTS)](https://uts.nlm.nih.gov/uts) and therefore enabling automated
download of UMLS and the following (non-exhaustive) list of resources:

- [RxNorm](https://www.nlm.nih.gov/research/umls/rxnorm/docs/rxnormfiles.html)
- [SemMedDB](https://lhncbc.nlm.nih.gov/ii/tools/SemRep_SemMedDB_SKR/SemMedDB_download.html)
- [SNOMED-CT](https://www.nlm.nih.gov/healthit/snomedct/international.html)
- potentially more in the future

This has been implemented in the open source
[`umls_downloader`{.language-plaintext
.highlighter-rouge}](https://github.com/cthoyt/umls-downloader) Python
package. It can be installed with the following one-liner in your shell:

:::: {.language-shell .highlighter-rouge}
::: {.highlight}
``` highlight
$ pip install umls_downloader
```
:::
::::

Below, I'll walk you through using it. Throughout, keep in mind that
full documentation for the `umls_downloader`{.language-plaintext
.highlighter-rouge} is available at
[umls-downloader.readthedocs.io](https://umls-downloader.readthedocs.io),
which describes the other functionality and other data that can be
downloaded.

### Usage

UMLS has three different distributions that are described
[here](https://www.nlm.nih.gov/research/umls/licensedcontent/umlsknowledgesources.html).
The following Python code downloads the most simple and straightforward
file, `MRCONSO.RRF`{.language-plaintext .highlighter-rouge} as a zip
archive:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from umls_downloader import download_umls

path = download_umls(version="2023AA", api_key="")
```
:::
::::

This code is smart and does not need to download the file more than
once. It uses [`pystow`{.language-plaintext
.highlighter-rouge}](https://github.com/cthoyt/pystow) to choose a
stable path `~/.data/bio/umls`{.language-plaintext .highlighter-rouge}
relative to the current user's home directory. Inside this directory, it
also uses the version of the data to create a subdirectory. Finally,
this function returns the path to the data, such that no file paths ever
need to be hard-coded.

> **Warning** This still requires an API key, which requires creating an
> account, agreeing to UMLS's terms and conditions, etc. This can be
> done here: <https://uts.nlm.nih.gov/uts/edit-profile>.

### Automating Configuration of UTS Credentials

There are two ways to automatically set the API key, so you don't have
to worry about getting it and passing it around in your python code:

1. Set `UMLS_API_KEY`{.language-plaintext .highlighter-rouge} in the
 environment. This can be done in your interactive session or in the
 configuration for your shell such as in a
 `.bashrc`{.language-plaintext .highlighter-rouge} file for the
 Bourne Again Shell (bash).
2. Create `~/.config/umls.ini`{.language-plaintext .highlighter-rouge}
 and set in the `[umls]`{.language-plaintext .highlighter-rouge}
 section a `api_key`{.language-plaintext .highlighter-rouge} key.
 Mine looks like:

 :::: {.language-ini .highlighter-rouge}
 ::: {.highlight}
 ``` highlight
 [umls]
 api_key=1234567890abcdefghijklmno
 ```
 :::
 ::::

Now you can omit the `api_key`{.language-plaintext .highlighter-rouge}
keyword like in the following:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from umls_downloader import download_umls

# Same path as before
path = download_umls(version="2023AA")
```
:::
::::

### Download the Latest Version

First, you'll have to install [`bioversions`{.language-plaintext
.highlighter-rouge}](https://github.com/cthoyt/bioversions) with
`pip install bioversions`{.language-plaintext .highlighter-rouge}, whose
job it is to look up the latest version of many databases. Then, you can
modify the previous code slightly by omitting the
`version`{.language-plaintext .highlighter-rouge} keyword argument:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from umls_downloader import download_umls

# Same path as before (when run on September 1st, 2023)
path = download_umls()
```
:::
::::

### Download and open the file

The UMLS file is zipped, so it's usually accompanied by the following
boilerplate code:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
import zipfile
from umls_downloader import download_umls

path = download_umls()
with zipfile.ZipFile(path) as zip_file:
 with zip_file.open("MRCONSO.RRF", mode="r") as file:
 for line in file:
 ...
```
:::
::::

This exact code is wrapped with the
`umls_downloader.open_umls()`{.language-plaintext .highlighter-rouge}
using Python's context manager, so it can more simply be written as:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
from umls_downloader import open_umls

with open_umls() as file:
 for line in file:
 ...
```
:::
::::

> **Note** The `version`{.language-plaintext .highlighter-rouge} and
> `api_key`{.language-plaintext .highlighter-rouge} arguments work the
> same for `umls_downloader.open_umls()`{.language-plaintext
> .highlighter-rouge} as in
> `umls_downloader.download_umls()`{.language-plaintext
> .highlighter-rouge}

At this point, it's up to you to decide how you want to consume the
`MRCONSO.RRF`{.language-plaintext .highlighter-rouge} file. Below, I
give a demo on how parsed this file in PyOBO in order to convert UMLS to
an OWL ontology.

## Why not an API?

The UMLS provides an
[API](https://documentation.uts.nlm.nih.gov/rest/home.html) for access
to tiny bits of data at a time. There are even two recent (last 5 years)
packages [`umls-api`{.language-plaintext
.highlighter-rouge}](https://pypi.org/project/umls-api)
[`connect-umls`{.language-plaintext
.highlighter-rouge}](https://pypi.org/project/connect-umls) that provide
a wrapper around them. However, API access is generally rate limited,
difficult to use in bulk, and slow. For working with UMLS (or any other
database, for that matter) in bulk, it's necessary to download full
database dumps.

## UMLS Conversions

Building on top of the automated download of UMLS, I implemented a
fit-for-purpose processor with the [PyOBO
framework](https://github.com/pyobo/pyobo) that converts UMLS into an
ontology (encoded either as OWL, OBO, or OBO Graph JSON) which can
therefore be used to generate semantic mappings in the SSSOM format. The
code that implements this can be found
[here](https://github.com/pyobo/pyobo/commit/be67eadf77afa270cf224cad71d5b1905ae78b53).
After installing PyOBO with `pip install pyobo`{.language-plaintext
.highlighter-rouge}, you can automatically download and convert UMLS
first into an ontology encoded in the [OBO flat file
format](https://owlcollab.github.io/oboformat/doc/GO.format.obo-1_4.html),
then convert to OWL with the following code. Note: you'll need
[`robot`{.language-plaintext
.highlighter-rouge}](https://robot.obolibrary.org/) for the second step:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
import pyobo

umls = pyobo.get_ontology("umls")

# Write simple OBO Format
umls.write_obo("umls.obo")

# Convert to OWL
from pyobo.utils.misc import obo_to_owl

obo_to_owl("umls.obo", "umls.owl")
```
:::
::::

In an ideal world, the results of such a conversion could be included as
a part of the [OBO Database
Ingestion](https://github.com/biopragmatics/obo-db-ingest), which
converts database resources available through PyOBO into ontology
artifacts, archives them on GitHub and Zenodo, and gives them PURLs all
on a weekly basis to make sure the most up-to-date version is available
as well as all previous named versions. Instead, we live in a world with
pineapple pizza and restrictive licenses.

One of the nice qualities of UMLS is that it is a semantic mapping hub.
It provides mostly complete mappings between many vocabularies including
[MeSH](https://bioregistry.io/mesh),
[NCIT](https://bioregistry.io/ncit),
[SNOMED-CT](https://bioregistry.io/snomedct),
[HPO](https://bioregistry.io/hpo),
[LOINC](https://bioregistry.io/loinc), and more. However, there are a
few caveats to consider:

1. UMLS mappings aren't all 1-to-1. For example, MeSH mappings
 typically include many UMLS terms (narrower) pointing to the same
 MeSH term (broader). For other vocabularies, such as NCBITaxon, UMLS
 mappings are more reliably 1-to-1. Thanks to Tiago Lubiana for
 pointing this out.
2. Mapping provenance is not available, so the
 `mapping_justification`{.language-plaintext .highlighter-rouge}
 field in SSSOM is uniformly filled with
 `sempav:UnspecifiedMatching`{.language-plaintext
 .highlighter-rouge}.
3. Similarly, UMLS does not apply precise semantic predicates for each
 mapping. This means that they are output in PyOBO and as SSSOM with
 `oboInOwl:hasDbXref`{.language-plaintext .highlighter-rouge} instead
 of more detailed types such as `skos:exactMatch`{.language-plaintext
 .highlighter-rouge}, `skos:narrowMatch`{.language-plaintext
 .highlighter-rouge}, and `skos:broaderMatch`{.language-plaintext
 .highlighter-rouge}. Tools like
 [Boomer](https://github.com/INCATools/boomer) can be used to address
 this (in part). The [Semantic Mapping Reasoning Assembler
 (SeMRA)](https://github.com/biopragmatics/semra) can also be
 configured with prior knowledge about UMLS mapping assumptions when
 aggregating and reasoning over semantic mappings at scale.

With that in mind, anything that can be loaded as an ontology in PyOBO
can also be exported with SSSOM, which I show below. For UMLS, this
looks like:

:::: {.language-python .highlighter-rouge}
::: {.highlight}
``` highlight
import pyobo

df = pyobo.get_sssom_df("umls", names=False)
df.to_csv("umls.sssom.tsv", sep="\t", index=False)
```
:::
::::

> **Note** You can set `names=True`{.language-plaintext
> .highlighter-rouge} to have PyOBO look up the names for all entities,
> but this is a bit of a rabbit hole since it requires getting and
> processing many external resources.

------------------------------------------------------------------------

There's much more to say about UMLS and SSSOM, but this is a good place
to pause and publish this post, since getting UMLS as SSSOM is a task a
lot of people have asked me for help with lately. I might also come back
and explain more about how I use the other resources from UMLS's UTS.

I've been working on improving reproducibility in the field of
cheminformatics for some time now. For example, I've written posts about
making data from [DrugBank](/2020/12/14/taming-drugbank.html) and
[ChEMBL](/2021/08/05/taming-chembl-sql.html) more actionable. Over the
last year, I've been preparing a concept with the editors of the
[Journal of Cheminformatics](https://jcheminf.biomedcentral.com) on how
to include an assessment of reproducibility to reviews of manuscripts
submitted to the journal. This has resulted in an editorial [Improving
reproducibility and reusability in the Journal of
Cheminformatics](https://doi.org/10.1186/s13321-023-00730-y) as well as
a [call for papers](https://www.biomedcentral.com/collections/irrijc).
In this post, I want to summarize the first generation review criteria
we developed, give an example of it applied in practice

## The Seven (First Generation) Criteria

There are many potential directions for reproducibility. Given the fact
that typical computational scientists are not trained as software
engineers, we decided on seven very simple criteria that can be easily
reviewed and easily addressed:

1.  Does the repository contain a LICENSE file in its root?
2.  Does the repository contain a README file in its root?
3.  Does the repository contain an associated public issue tracker?
4.  Has the repository been externally archived on
    [Zenodo](https://zenodo.org/), [FigShare](https://figshare.com/), or
    equivalent that is referenced in the README?
5.  Does the README contain installation documentation?
6.  Is the code from the repository installable in a straight-forward
    manner?
7.  Does the code conform to an external linter (e.g.,
    [`black`{.language-plaintext
    .highlighter-rouge}](https://github.com/psf/black) for Python)?

These correspond to important details that are complementary to other
considerations of reproducibility, but often overlooked. Throughout the
pilot, the editors and reviewers will try to support authors in
addressing each of these points during revision. I imagine that there
will be future iterations of these criteria as the community begins to
expect these as standard practice. For example, we can narrow criteria 1
to specifically say that the software should be licensed with an
OSI-approved license and not accept science made with non-open licenses.
We could further narrow point 7 to have additional community style
requirements (e.g., passes parts of `flake8`{.language-plaintext
.highlighter-rouge}, as you know I love from my post [on flake8
hell](/2020/04/25/how-to-code-with-me-flake8.html)). We could also
include additional guidelines that e.g. say that the results presented
in the paper should be reproducible with a single command from the
command line, e.g., a shell script. The rabbit hole could go very deep,
so again, it's worth saying that these are very non-controversial
criteria for the first generation.

That being said, many repositories don't follow these! Since these
criteria are so simple, I'm interested in automating their assessment
and further applying it to the entire Journal of Cheminformatics
backlog. I'll describe this more in a future post.

Without further ado, the text below is what I sent verbatim in the
review for [Drug-Protein Interaction Prediction via Multi-View
Variational Autoencoder and Cascade Deep
Forests](https://doi.org/10.21203/rs.3.rs-3242419/v1), which is
pre-printed on Research Square and has associated code
[here](https://github.com/Macau-LYXia/MVAE-DFDTnet). I have tried my
best to include actionable links and information with each piece. I
would like to also automate sending separate GitHub issues for each of
these points as a more concrete to-do list for authors, then also send
an "epic" issue that lists all of them together. With the magic of the
GitHub API, this is possible.

## My First Reproducibility Review for the Reproducibility Pilot

Below, I apply the seven point reproducibility review prescribed by
[Improving reproducibility and reusability in the Journal of
Cheminformatics](https://doi.org/10.1186/s13321-023-00730-y) to the
default branch of repository https://github.com/Macau-LYXia/MVAE-DFDTnet
(commit
[c0858c8](https://github.com/Macau-LYXia/MVAE-DFDTnet/commit/c0858c8cdf61d62f945fbd74d0d60f2531394287)),
accessed on August 27^th^, 2023.

1.  **Does the repository contain a LICENSE file in its root?**  
    No. The GitHub license picker can be used to facilitate adding one
    by following this link:
    https://github.com/Macau-LYXia/MVAE-DFDTnet/community/license/new?branch=main.
    Ideal software licenses for open source software include the MIT
    License, BSD family of licenses, and other licenses approved by the
    [Open Source Initiative](https://opensource.org/licenses). A simple,
    informative guide for picking a license can be found at
    https://choosealicense.com.

2.  Does the repository contain a README file in its root?  
    No. A minimal viable README file contains:

    - A short, one line description of the project
    - Information on how to download, install, and run the code locally
    - Brief documentation describing the single most important use case
      for the repository. For scientific code, this is ideally a
      one-liner in Python code, a shell script, or a command line
      interface (CLI) that can be used to reproduce the results of the
      analysis presented in a corresponding manuscript, use the tool
      presented in the manuscript, etc.
    - Link to an archive on an external system like Zenodo, FigShare, or
      an equivalent.
    - Citation information, e.g., for a pre-print then later for a peer
      reviewed manuscript

    GitHub can be used to create a README file with
    https://github.com/Macau-LYXia/MVAE-DFDTnet/new/main?filename=README.md.
    Repositories typically use the Markdown format, which is explained
    [here](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax).

3.  Does the repository contain an associated public issue tracker?  
    Yes. It is available at
    https://github.com/Macau-LYXia/MVAE-DFDTnet/issues.

4.  Has the repository been externally archived on Zenodo, FigShare, or
    an equivalent that is referenced in the README?  
    No, there is no README. This is also not mentioned in the
    manuscript. See
    https://docs.github.com/en/repositories/archiving-a-github-repository/referencing-and-citing-content
    for a tutorial on how to do this.

5.  Does the README contain installation documentation?  
    No, there is no README. This is also not mentioned in the
    manuscript.

6.  Is the code in the repository installable in a straight-forward
    manner?  
    No, the code is not laid out in a typical structure, e.g., as
    described in https://blog.ionelmc.ro/2014/05/25/python-packaging.
    Further, there is no setup configuration that encodes the
    dependencies or facilitates reuse of the code.

7.  Does the code in the repository conform to an external linter (e.g.,
    `black`{.language-plaintext .highlighter-rouge} for Python)?  
    No. The Python code has not been linted, e.g., using
    [`black`{.language-plaintext
    .highlighter-rouge}](https://github.com/psf/black). Similarly, the
    Matlab code has not been linted, e.g. using
    [`checkcode`{.language-plaintext
    .highlighter-rouge}](https://de.mathworks.com/help/matlab/ref/checkcode.html).

Scientific integrity depends on enabling others to understand the
methodology (written as computer code) and reproduce the results
generated from it. This reproducibility review reflects steps towards
this goal that may be new for some researchers, but will ultimately
raise standards across our community and lead to better science. Because
the work presented in this article only yet address one of the seven
points of the reproducibility review, I recommend rejecting the article
and inviting later resubmission following addressing the points.

For posterity, this review has also been included on
https://github.com/Macau-LYXia/MVAE-DFDTnet/issues/1.

## The Future is Looking Good

The example above isn't so great - it's possible that these authors have
never considered most of these points about reproducibility before. The
reality is that many computational scientists are not trained in this
since their mentors were not primarily trained as computational
scientists themselves. Combine with the perverse incentive structure in
academia, it's understandable how this can be left out from some
publications. I experienced something similar in my doctoral studies,
and had to bootstrap my own philosophy on reproducibility as well as the
practical skills to achieve it. I also understand not everyone is in the
position where they have the flexibility/freedom/initiative to do this.

That all being said, we are now entering an era where progressive and
newly minted PIs actually have training as computational scientists. The
next paper in my queue for a reproducibility review is for
https://github.com/Steinbeck-Lab/cheminformatics-python-microservice,
which will pass the 7 criteria with flying colors. I'm looking forward
to the future when we expect more excellent science on the regular. See
you there!

------------------------------------------------------------------------

I'm not sure how people will view the way I talk about reviews - I am
quite open with posting reviews on GitHub and also openly discussing the
fact that I've reviewed something. Ideally, I don't accept reviews for
papers that don't have pre-prints, since I personally think the review
process should be open. I hope it's the case that I haven't been rude or
unfair. If that's the case, someone can help me change the way I write
about these topics.
