Charles Tapley Hoyt

I've been working on improving reproducibility in the field of
cheminformatics for some time now. For example, I've written posts about
making data from [DrugBank](/2020/12/14/taming-drugbank.html) and
[ChEMBL](/2021/08/05/taming-chembl-sql.html) more actionable. Over the
last year, I've been preparing a concept with the editors of the
[Journal of Cheminformatics](https://jcheminf.biomedcentral.com) on how
to include an assessment of reproducibility to reviews of manuscripts
submitted to the journal. This has resulted in an editorial [Improving
reproducibility and reusability in the Journal of
Cheminformatics](https://doi.org/10.1186/s13321-023-00730-y) as well as
a [call for papers](https://www.biomedcentral.com/collections/irrijc).
In this post, I want to summarize the first generation review criteria
we developed, give an example of it applied in practice

## The Seven (First Generation) Criteria

There are many potential directions for reproducibility. Given the fact
that typical computational scientists are not trained as software
engineers, we decided on seven very simple criteria that can be easily
reviewed and easily addressed:

1.  Does the repository contain a LICENSE file in its root?
2.  Does the repository contain a README file in its root?
3.  Does the repository contain an associated public issue tracker?
4.  Has the repository been externally archived on
    [Zenodo](https://zenodo.org/), [FigShare](https://figshare.com/), or
    equivalent that is referenced in the README?
5.  Does the README contain installation documentation?
6.  Is the code from the repository installable in a straight-forward
    manner?
7.  Does the code conform to an external linter (e.g.,
    [`black`{.language-plaintext
    .highlighter-rouge}](https://github.com/psf/black) for Python)?

These correspond to important details that are complementary to other
considerations of reproducibility, but often overlooked. Throughout the
pilot, the editors and reviewers will try to support authors in
addressing each of these points during revision. I imagine that there
will be future iterations of these criteria as the community begins to
expect these as standard practice. For example, we can narrow criteria 1
to specifically say that the software should be licensed with an
OSI-approved license and not accept science made with non-open licenses.
We could further narrow point 7 to have additional community style
requirements (e.g., passes parts of `flake8`{.language-plaintext
.highlighter-rouge}, as you know I love from my post [on flake8
hell](/2020/04/25/how-to-code-with-me-flake8.html)). We could also
include additional guidelines that e.g. say that the results presented
in the paper should be reproducible with a single command from the
command line, e.g., a shell script. The rabbit hole could go very deep,
so again, it's worth saying that these are very non-controversial
criteria for the first generation.

That being said, many repositories don't follow these! Since these
criteria are so simple, I'm interested in automating their assessment
and further applying it to the entire Journal of Cheminformatics
backlog. I'll describe this more in a future post.

Without further ado, the text below is what I sent verbatim in the
review for [Drug-Protein Interaction Prediction via Multi-View
Variational Autoencoder and Cascade Deep
Forests](https://doi.org/10.21203/rs.3.rs-3242419/v1), which is
pre-printed on Research Square and has associated code
[here](https://github.com/Macau-LYXia/MVAE-DFDTnet). I have tried my
best to include actionable links and information with each piece. I
would like to also automate sending separate GitHub issues for each of
these points as a more concrete to-do list for authors, then also send
an "epic" issue that lists all of them together. With the magic of the
GitHub API, this is possible.

## My First Reproducibility Review for the Reproducibility Pilot

Below, I apply the seven point reproducibility review prescribed by
[Improving reproducibility and reusability in the Journal of
Cheminformatics](https://doi.org/10.1186/s13321-023-00730-y) to the
default branch of repository https://github.com/Macau-LYXia/MVAE-DFDTnet
(commit
[c0858c8](https://github.com/Macau-LYXia/MVAE-DFDTnet/commit/c0858c8cdf61d62f945fbd74d0d60f2531394287)),
accessed on August 27^th^, 2023.

1.  **Does the repository contain a LICENSE file in its root?**  
    No. The GitHub license picker can be used to facilitate adding one
    by following this link:
    https://github.com/Macau-LYXia/MVAE-DFDTnet/community/license/new?branch=main.
    Ideal software licenses for open source software include the MIT
    License, BSD family of licenses, and other licenses approved by the
    [Open Source Initiative](https://opensource.org/licenses). A simple,
    informative guide for picking a license can be found at
    https://choosealicense.com.

2.  Does the repository contain a README file in its root?  
    No. A minimal viable README file contains:

    - A short, one line description of the project
    - Information on how to download, install, and run the code locally
    - Brief documentation describing the single most important use case
      for the repository. For scientific code, this is ideally a
      one-liner in Python code, a shell script, or a command line
      interface (CLI) that can be used to reproduce the results of the
      analysis presented in a corresponding manuscript, use the tool
      presented in the manuscript, etc.
    - Link to an archive on an external system like Zenodo, FigShare, or
      an equivalent.
    - Citation information, e.g., for a pre-print then later for a peer
      reviewed manuscript

    GitHub can be used to create a README file with
    https://github.com/Macau-LYXia/MVAE-DFDTnet/new/main?filename=README.md.
    Repositories typically use the Markdown format, which is explained
    [here](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax).

3.  Does the repository contain an associated public issue tracker?  
    Yes. It is available at
    https://github.com/Macau-LYXia/MVAE-DFDTnet/issues.

4.  Has the repository been externally archived on Zenodo, FigShare, or
    an equivalent that is referenced in the README?  
    No, there is no README. This is also not mentioned in the
    manuscript. See
    https://docs.github.com/en/repositories/archiving-a-github-repository/referencing-and-citing-content
    for a tutorial on how to do this.

5.  Does the README contain installation documentation?  
    No, there is no README. This is also not mentioned in the
    manuscript.

6.  Is the code in the repository installable in a straight-forward
    manner?  
    No, the code is not laid out in a typical structure, e.g., as
    described in https://blog.ionelmc.ro/2014/05/25/python-packaging.
    Further, there is no setup configuration that encodes the
    dependencies or facilitates reuse of the code.

7.  Does the code in the repository conform to an external linter (e.g.,
    `black`{.language-plaintext .highlighter-rouge} for Python)?  
    No. The Python code has not been linted, e.g., using
    [`black`{.language-plaintext
    .highlighter-rouge}](https://github.com/psf/black). Similarly, the
    Matlab code has not been linted, e.g. using
    [`checkcode`{.language-plaintext
    .highlighter-rouge}](https://de.mathworks.com/help/matlab/ref/checkcode.html).

Scientific integrity depends on enabling others to understand the
methodology (written as computer code) and reproduce the results
generated from it. This reproducibility review reflects steps towards
this goal that may be new for some researchers, but will ultimately
raise standards across our community and lead to better science. Because
the work presented in this article only yet address one of the seven
points of the reproducibility review, I recommend rejecting the article
and inviting later resubmission following addressing the points.

For posterity, this review has also been included on
https://github.com/Macau-LYXia/MVAE-DFDTnet/issues/1.

## The Future is Looking Good

The example above isn't so great - it's possible that these authors have
never considered most of these points about reproducibility before. The
reality is that many computational scientists are not trained in this
since their mentors were not primarily trained as computational
scientists themselves. Combine with the perverse incentive structure in
academia, it's understandable how this can be left out from some
publications. I experienced something similar in my doctoral studies,
and had to bootstrap my own philosophy on reproducibility as well as the
practical skills to achieve it. I also understand not everyone is in the
position where they have the flexibility/freedom/initiative to do this.

That all being said, we are now entering an era where progressive and
newly minted PIs actually have training as computational scientists. The
next paper in my queue for a reproducibility review is for
https://github.com/Steinbeck-Lab/cheminformatics-python-microservice,
which will pass the 7 criteria with flying colors. I'm looking forward
to the future when we expect more excellent science on the regular. See
you there!

------------------------------------------------------------------------

I'm not sure how people will view the way I talk about reviews - I am
quite open with posting reviews on GitHub and also openly discussing the
fact that I've reviewed something. Ideally, I don't accept reviews for
papers that don't have pre-prints, since I personally think the review
process should be open. I hope it's the case that I haven't been rude or
unfair. If that's the case, someone can help me change the way I write
about these topics.


I’ve been working on improving reproducibility in the field of cheminformatics for some time now. For example, I’ve written posts about making data from DrugBank and ChEMBL more actionable. Over the last year, I’ve been preparing a concept with the editors of the Journal of Cheminformatics on how to include an assessment of reproducibility to reviews of manuscripts submitted to the journal.

Reproducibility Pilot in the Journal of Cheminformatics

Imperial College London

Henry Rzepa

This is one of those posts of a molecule whose very structure is
interesting enough to merit a picture and a 3D model. The
study\[cite\]10.1126/science.aal1619\[/cite\] reports a molecular knot
with the remarkable number of eight crossings.

[![](http://www.ch.ic.ac.uk/rzepa/blog/wp-content/uploads/2017/01/085-1021x1024.jpg){.aligncenter
.size-large .wp-image-17367 loading="lazy" decoding="async" width="450"
height="451"
srcset="https://www.ch.ic.ac.uk/rzepa/blog/wp-content/uploads/2017/01/085-1021x1024.jpg 1021w, https://www.ch.ic.ac.uk/rzepa/blog/wp-content/uploads/2017/01/085-150x150.jpg 150w, https://www.ch.ic.ac.uk/rzepa/blog/wp-content/uploads/2017/01/085-300x300.jpg 300w, https://www.ch.ic.ac.uk/rzepa/blog/wp-content/uploads/2017/01/085-768x770.jpg 768w, https://www.ch.ic.ac.uk/rzepa/blog/wp-content/uploads/2017/01/085-144x144.jpg 144w, https://www.ch.ic.ac.uk/rzepa/blog/wp-content/uploads/2017/01/085.jpg 1093w"
sizes="(max-width: 450px) 100vw, 450px"}](https://doi.org/10.5517/CCDC.CSD.CC1M85Y0){target="new"}

The DOI for the 3D model is
[10.5517/CCDC.CSD.CC1M85Y0](https://doi.org/10.5517/CCDC.CSD.CC1M85Y0){target="new"} (or
click on the image above). Such topology intersects with work we did a
few years back on high-order crossings in fully
conjugated π-systems\[cite\]10.1021/ja710438j\[/cite\], which were then
illustrated\[cite\]10.1021/jp902176a\[/cite\] with hypothetical charged
higher order annulenes exhibiting linking numbers Lk of up to 6π. A
fully π-conjugated system, also with a linking number in the
π-framework of 6π but in the form of a trefoil braid was even suggested
on [this blog](http://www.ch.imperial.ac.uk/rzepa/blog/?p=2046), with a
common feature of a central templating atom (a cation rather than an
anion). Another example of a previously reported pentadecanuclear
manganese metallacycle\[cite\]10.1021/ic800987f\[/cite\] was also
assigned a linking number of 6π.

The molecule above is not completely π-conjugated around the braid and
so special properties related to aromaticity and associated [ring
currents](http://www.ch.imperial.ac.uk/rzepa/blog/?p=9322) resulting
from the topology of the cyclic
conjugation\[cite\]10.1016/j.comptc.2014.09.028\[/cite\] are not
expected to accrue in the eight-crossing molecular
braid\[cite\]10.1126/science.aal1619\[/cite\]. We might also look
forward to examples of the characterisation of braids with an
**odd-number** of crossings such as trefoils, pentafoils, heptafoils,
etc, as associated with the
name [Möbius](http://www.ch.imperial.ac.uk/rzepa/blog/?p=3492).

::: {.wp_orcid_field}
[https://orcid.org/0000-0002-8635-8390](https://orcid.org/0000-0002-8635-8390){target="_blank"
rel="author"}
:::


This is one of those posts of a molecule whose very structure is interesting enough to merit a picture and a 3D model. The study[cite]10.1126/science.aal1619[/cite] reports a molecular knot with the remarkable number of eight crossings.

Braiding a molecular knot with eight crossings.

Peter Murray-Rust and I are delighted to announce that the 2016 award of
the [Bradley-Mason prize for open
chemistry](http://www.ch.imperial.ac.uk/rzepa/blog/?p=14224) goes to Jan
Szopinski (UG) and Clyde Fare (PG).

Jan's open chemistry derives from a final year project looking at why
atom charges derived from quantum chemical calculation of the electronic
density represent chemical information well, but the electrostatic
potential (ESP) generated from these charges is very poor and
conversely charges derived from the computed electrostatic potential are
incommensurate with chemical information (such as the electronegativity
of atoms). He has developed a Python program called '**repESP**' in
which 'compromise' charges are generated which attempt to reconcile the
physical world-view (fitting the ESP) with chemical insight provided by
NPA (Natural Population Analysis). Jan was the main driver to making his
code open source, "opening his supervisor's eyes" to the various
flavours of open source licences. To ensure that all subsequent
improvements to the program remain available to anyone, the source code
has been released under a 'copyleft' licence (GPL v3) and is maintained
by Jan on [GitHub](https://github.com/jszopi/repESP), where Jan looks
forward to helping new users and collaborating with contributors.

Clyde has made various contributions to opensource chemistry over the
period of his PhD, with the focus mainly on utilities to improve quantum
chemical research and the enhancement of a popular machine learning
library with a method that has been successful in chemometrics, creation
of an opensource channel for teaching chemists programming and data
analysis and creation of a tool to help encourage open sourcing software
development. [Cclib](http://cclib.github.io) is the most popular library
for parsing quantum chemical data from output files and Clyde has
contributed patches for the [Atomic simulation
environment](https://gitlab.com/ase/ase) which enables control of
quantum chemical codes from a unified python interface. He was
responsible for the construction of a [computational chemistry
electronic notebook](https://github.com/Clyde-fare/cc_notebook)
published to github and which is now under active development by others
as well. This aims to encapsulate computation chemical research
projects, both for the sake of reproducibility and for the sake of
organising and keeping track of quantum chemical research. Alongside
this platform he created an enhanced Gaussian calculator for the Atomic
Simulation Environment that enables automatic construction of ONIOM
input files, also now under [active
development](https://github.com/Clyde-fare/gausspy). He also made
[contributions to scikit
learn](https://github.com/scikit-learn/scikit-learn/blob/b8be0198b6b66a67721a19d247a8343486578b73/doc/whats_new.rst),
the most popular python machine learning framework, implementing a
kernel for Kernel Ridge Regression that has become the most successful
kernel for regression over molecular properties. He was part of the team
that won the 2014 sustainable software conference prize for creation of
the [opensource healthchecker
software](https://github.com/OpenSourceHealthCheck) as part of Sustain.
He has argued for opensource as a platform for teaching resources and
created the [Imperial Chemistry github](https://github.com/imperialchem)
user account, which is now run by the department. Materials for the
*Imperial Chemistry Data Analysis and Programming* workshops implemented
as Python Notebooks are now available through this account and continue
under active development.

Criteria for the award will include judging the submission on its
immediate accessibility *via* public web sites, what is visible and
re-usable in this way and of evidence of either community
formation/engagement or re-use of materials by people other than the
proposer.

::: {.wp_orcid_field}
[https://orcid.org/0000-0002-8635-8390](https://orcid.org/0000-0002-8635-8390){target="_blank"
rel="author"}
:::


Peter Murray-Rust and I are delighted to announce that the 2016 award of the Bradley-Mason prize for open chemistry goes to Jan Szopinski (UG) and Clyde Fare (PG).  Jan’s open chemistry derives from a final year project looking at why atom charges derived from quantum chemical calculation of the electronic density represent chemical information well, but the electrostatic potential (ESP) generated from these charges is very

Messaggi di Rogue Scholar

Reproducibility Pilot in the Journal of Cheminformatics

Braiding a molecular knot with eight crossings.

The 2016 Bradley-Mason prize for open chemistry.