Thoughts about bibliographic metadata, programming, statistics, taxonomy, and biology.

Lars Willighagen

I’m going to share some stories on debugging with you, because I’m proud of them. After writing up the first story, I’m no longer particularly proud, but I still want to share the story. Here’s the first: a bug that seemed quite familiar.  After some trouble with mocking API requests I decided that supporting mocking in the browser isn’t as important as supporting mocking at all, so I installed mock-require (I didn’t get proxyquire to work).

Debugging: Deja Vu

Since the citation-js npm package was first published, version 0.6 is the first major version of Citation.js that did not start out as a pre-release. Version 0.3 itself spent almost 6 months in pre-release, but only received updates for less than half a month. Version 0.4 spent more than a year in pre-release and received updates for about 4 months.

Citation.js Version 0.6: CSL v1.0.2

The new update (Citation.js v0.2.10) doesn't have a big impact on the API, but a lot has changed in the back end. ("back end" as in the helper functions that are called when the API is used.) First of all, there is Travis build tests now. It doesn't have edge test cases yet, but it covers the basics. Testing the basic specs  The main thing is the new BibTeX I/O system. It isn't completely new, but a lot has changed.

Citation.js Version 0.2.10: BibTeX and Travis

Earlier this week, tarrow published factvis, short for fact visualisation. I decided to have a go with the design, and I made this, in the style of cardlists.

 Note:

If my version and tarrow's version of factvis look very similar, my changes are probably pushed to the master branch already. Screenshot of my factvis design The facts being visualised come from the ContentMine.

Weekly Report 10: Visualising facts and asking questions

Working on translating a key to the European shield bug nymphs (Puchkov, 1961) I thought I would look for pictures of the earlier life stages (nymphs, Fig. 1) of shield bugs (Pentatomoidea) on iNaturalist and found few observations actually had the life stage annotation.

Finding shield bug nymphs on iNaturalist

Last november I wrote a blog post about how to model the taxonomic coverage of identification keys. I wanted to model this coverage to be able to determine to what extent an identification key applies to a given observation or specimen, for use in my Library of Identification Resources project. For the same project I also find it useful to be able to archive identification keys.

On the modelling of the content of identification keys

The CDLI is developing a new website. That website’s admin interface for its journals contains a page where a LaTeX source file, following a specific template, is configured to an HTML page. For this, apart from the LaTeX file itself, two additional components are needed: a BibTeX file, containing metadata of the references; and image files.

Re-implementing the upload of images for the LaTeX→HTML converter

I recently found myself working with Altmetric badges again, and I realized how cumbersome it can be to work with scripts. The Altmetric badges can only be added by using their JavaScript library, while it would be a lot more user friendly to have a simple URL that embeds the badge, preferably even an image.

Modern Altmetric badges

It’s been a while. Really. But now it’s back, with a new release: v0.4.0-0, the v0.4 beta. Below I explain some of the changes in this release, and then the road-map of Citation.js for v0.4 and v0.5. Also, Citation.js has a DOI now:    Also, @jsterlibs tweeted about Citation.js. Release  Input plugins   The main change in this release, is the addition of input plugins. This is the first step towards releasing v0.4, as explained below.

Citation.js Version 0.4 Beta: New Docs and Input Plugins

Citation.js now supports BibJSON. How I did that without actually updating Citation.js? Well, apparently I supported it all along. I've supported the quickscrape output format since July last year, and that turned out to be BibJSON. How convenient. I'll update the demo and docs to reflect this revelation (currently it just says "quickscrape's JSON scheme"), and, now that I can find actual documentation, some improvements to the parser.

Citation.js: BibJSON

When developing Citation.js, I needed to get a JavaScript Object from a string representing one. There are several methods for this, and the one that comes to mind first is JSON.parse(). However, this didn't work.


Parsing invalid JSON

This weekly report covers the past two weeks. I blogged twice last week, and I figured that was enough. Last week I blogged about word clouds from ContentMine output. I also blogged about ctj. This week, I have combined both into interactive lists, as seen here and in the images below. List overview. From left to right: articles, and genus/genera and species that were mentioned in the articles. Search results.


Weekly Report 7: Interactive ContentMine output

Yesterday I published a blogpost, where I talked about

 ctj

and how and why to convert ContentMine's CProjects to JSON. At the end, I mentioned this post, where I would talk about how to use it in different programs, and with d3.js. So here we go. For starters, let's make the data about word frequencies look nice. Not readable (then we would use a table), but visually pleasing. Let's make a word cloud.

Word Clouds

Originally posted on the ContentMine blog

I am Lars, and I am from the Netherlands, where I currently live. I applied to this fellowship to learn new things and combine the ContentMine with two previous projects I never got to finish, and I got really excited by the idea and the ContentMine at large.

Introducing Fellow Lars Willighagen: Constructing and visualising a network-based conifer database

Citation.js has supported Wikidata input for a long time. However, I’ve always had some trouble with the API.

Citation.js: Use Case for a Wikidata GraphQL API

A while back I tweeted about making a simple Citation.js API Endpoint with RunKit.

Citation.js: Endpoint on RunKit

Version 0.7.9 of Citation.js comes with a new feature: plugin-bibtex now supports the import and export of Data Annotations in BibLaTeX files. This means ORCID identifiers from DOI, Wikidata, CFF, and other sources can now be exported to BibLaTeX. Combined with a BibLaTeX style that displays ORCID identifiers, you can now quickly improve your reference lists with ORCIDs.

Citation.js: BibLaTeX Data Annotations support

Today I worked on three user scripts for Wikidata. Together, these tools hopefully make the data in Wikidata more accessible and make it easier to navigate between items. To enable these, include one or more of the following lines in your common.js (depending on which script(s) you want): mw.loader.load('//www.wikidata.org/w/index.php?title=User:Lagewi/properties.js&amp;oldid=2039401177&amp;action=raw&amp;ctype=text/javascript');

Three new userscripts for Wikidata

Ever since I found out about Structured Descriptive Data (SDD) I keep coming back to it. Right now I am planning to make two identification keys, one translated and modernized from a 1961 publication for which I have not found a good alternative, and one completely

 de novo

(well, nearly). In both cases, I started writing out the key in a word processor with a numbered list and tabs and manual alignment etc., but in both cases that

Ingesting Structured Descriptive Data into Pandoc

Following up on the previous two updates this year (

 Version 0.5 and a 2022 update

and

 Version 0.6: CSL v1.0.2

), here are some updates from the second half of 2022, as well as some statistics.

 Sapygina decemguttata

, a small wasp, observed July 8th, 2021 Changes The mappings of the Wikidata plugin were updated, especially to accomodate for software. The default-locale setting of some CSL styles is now respected.

Citation.js: 2022 in review

The website of the Library of Identification Resources has a new feature: a map view. The resources in the catalog are associated with a geographic scope to approximate which species in a taxonomic group can be identified. These geographic scopes are usually countries or continents but can also be subdivisions within countries, multinational ecological regions like mountain ranges, and biogeographical realms.

Explore identification keys on the world map

Since around this time last year, I have been working on creating a library of identification resources. Here, “identification resources” are identification keys, multi-access (matrix) keys, other works that can aid in the identification of species. The project is managed on GitHub: https://github.com/identification-resources.

Library of Identification Resources

Last semester I followed some courses at a different university, and went through the process of collecting login credentials and multi-factor authentication tokens and familiarizing myself with a network of university systems all over again.

A story about a university login with a broken security configuration, and a mildly uncooperative help desk

I recently updated the website of Citation.js. This involved getting rid of the Material Design Lite framework, simplifying and refreshing the site design and modernisering some of the code behind it. Additionally, I updated the content of the homepage, and added some functionality to the interface of the blog page and the demo.

Citation.js: New site

Since Google Summer of Code 2020 I have been contributing code to the new framework of the Cuneiform Digital Library Initiative (CDLI), a digital repository for metadata, transliterations, images, and other data of cuneiform inscriptions, as well as tools to work with that data. One of the features of the new CDLI framework is improved editing of artifact metadata, as well as inscriptions.

CDLI catalogue growth over time

Recently I moved the Citation.js API documentation from /api to /api/0.3, to put the new documentation on /api/0.5. I fixed all the links to the documentation, but I still got a issue request regarding a 404 error after just a few days. All in all, I had to redirect pages from /api/* to /api/0.3/* while all these pages are hosted as static files on GitHub Pages.

GitHub pages 404 redirection

Every week since the start of January I heard headlines like “3963 new infections, a bit more than average”. In context, that meant that the amount of positive COVID-19 that day was higher than the average daily amount of cases in the previous seven days. Even though we heard that every week, overall the cases were going down.

Mid-week effect of Dutch COVID-19 case reporting

A rework of the BibTeX parser has been on the backlog since at least August 15, 2017, and recently I started working on actually carrying it out — systematically. There were a number of things to be improved:

 Complete syntax support:

again, supporting BibTeX by looking at examples leads in a lack of support for less seen features like @string, @preamble and parentheses for enclosing entries instead of braces.

BibTeX Rework: Syntax Update

For reasons I will not go through right now, I needed a new library for making synchronous HTTP requests in Node.js. I know what you are saying, “But that’s one of the seven deadly sins of JavaScript!”

 1

Well, just know I had my reasons, and I wanted to replace sync-request.

Debugging the Karmabug

In the last post I explained how I started implementing the RIS specification that I found in the Internet Archive, only to discover that there is an older specification, which seems to be more common at times. Now, I have implemented the old spec, which luckily was not nearly as complex.

Citation.js: RIS Rework Pt. 2

So a while ago I was looking around for the RIS specification again. I had not found it earlier, only a reference implementation from Zotero, a surprisingly complete list of tags and types on Wikipedia and some examples from various websites and programs exporting RIS files. They did not seem to go together well, however.

Citation.js: RIS Rework Pt. 1

Citation.js just had a bunch of tooling updated, which should make a lot of use cases easier. Let’s go through them: Replacer (GitHub, Demo)

 Replacer

is an HTML API for Citation.js that I recently updated for the new version so that it works with components.

Citation.js: Usability Update

The new update, v0.4.4, contains a few Wikidata improvements (commit): 22 new mappings 2 fixed mappings (ISSN did not work and publisher-place was mapped to the wrong thing) 2 improved mappings (container-title for chapters and more URL mappings) 1 removed mapping (genre was inconsistent with the intended use, although it followed the specification)  Because most of these mappings require additional resources (recipient has to fetch people,

Citation.js: Wikidata Update

After about two years, finally v0.4.0 is ready for a release. Whether it is a real milestone given that there have been prereleases for about two years, while 0.3.x only lasted three weeks, is for you to decide but I am glad that everything I planned for this version is implemented now. Changes   So, let’s go through the major changes. Modules   Most importantly, the code is fully modulated now.

Citation.js Version 0.4: The Catch-up

I am in the process of creating a better way to map CSL types to Wikidata types together with Jakob Voß, and while listing all subtypes of web page, I came across a number of these cases (link): #defaultView:Graph SELECT ?a ?aLabel ?b ?bLabel WHERE {   wd:Q58803899 wdt:P279* ?a .   ?a wdt:P279 ?b .   FILTER EXISTS {     ?b wdt:P279* wd:Q386724 .   }   SERVICE wikibase:label { bd:serviceParam wikibase:language "en". } }    I’m no expert on

Citation.js: Wikidata Subclasses

I made a Material-themed page showing Citation.js blog posts from Blogger. It supports pagination, tags, search and linking individual posts. Since it’s a single, static page I can’t support meta and link tags for metadata, that would require JavaScript which indexers don’t run.

Citation.js: Showing Blogger Posts on a Different Site

I finished the General Plugin system for Citation.js a few days ago (more on that later), so I could finally publish a new beta release. Now, after that half-finished piece of code had been blocking other work for a long while, I can at last start… fixing bugs, and closing other items in the backlog.  One of the items that has been on the backlog for a long time, and was on the backlog of the previous major version too, was sorting out BibJSON.

Journal Metadata: Authors &amp; Institutions

It’s been in beta since January 30, but here it is: Citation.js version 0.3. Below I explain the changes since the last blog post, and under that is a more complete change log. Also some upcoming changes.

Citation.js Version 0.3 Released!

Originally posted on the ContentMine blog.

Lars Willighagen, orcid:0000-0002-4751-4637 Final Report of my fellowship at the ContentMine. Proposal My proposal was to extract facts about various conifer species by analysing text from papers with software suited for analysing text and the tools provided by the ContentMine. These facts were then to be converted into JSON, and then viewable with an HTML (+CSS/JS) interface.

Final Report: Analysing and visualising data from papers about conifers

I promised more photos in the previous post, so here they are.  Penicillium: Detail With conidiophores Detail of conidiophores  More fungi:      Cat brains:    This post is part of a series.

Microscopic photography: Part 2

I got to make photographs of pre-made specimens with a microscope, and I wanted to show some of them. This is part one of a longer series; there were a lot of specimens.  Below are details of the cross section of a basswood stem. Detail Outer layer Center  Human artery tissue.      Human bone tissue (with some unidentified dark stuff).      More next week!

Microscopic photography: Part 1

My new homepage is finally finished (for now). I say finally, because it has taken a while before it actually felt done. You know how the first 90% of the work takes 90% of the time, but the last 10% of the work takes another 90% of the time? I just had to finish writing a single, small piece of text and it would be complete, but it took about 2 weeks.

New homepage design: New content, Material Design, Angularjs and more

When I started programming a few years ago, one of my first projects was linked to a project for school. Our team was making a presentation, and at some point I decided I wanted to program it myself, for the web. This resulted in the version of a project which, to this day, doesn't have a name (but soon will have).  After a few months I wanted to expand this.

SVM: Developing a brand new 3D model language for the Web

A few weeks ago I published version 0.2.11 of Citation.js. The main change was the addition of jQuery.Citation.js, updated for version 2 of Citation.js. jQuery.Citation.js is a small jQuery plugin to build simple forms where you can fill in metadata, which gets translated to CSL-JSON. The configuration options are currently quite limited, so it only really works as a demo for Citation.js itself.

Citation.js Version 0.2.11: jQuery and Looking Forward

This week, the big achievement is the addition of a multi-step form to add the semantic triples from last week to Wikidata with QuickStatements, which we talked about before too. The new '+' icon in table rows now links to a page where you can curate the statement and add Wikidata IDs where necessary. At the last step, you get a table of the existing data, the added identifiers and soon their Wikidata label.

Weekly Report 12: Including table facts in Wikidata

In Weekly Report 10 I talked about searching for answers to the question "What height does a grown

 Pinus sylvestris

normally have?". In the post, I looked at some of the articles returned by the query "Pinus sylvestris"[Abstract] AND height, and found interesting information in the tables. The next step was to extract this information. So that's what I did.

Weekly Report 11: Making Object-Property-Value triples

In the last two weeks I've been busy with making Version 0.2 of Citation.js. Here I'll explain some of the changes and the reasoning behind them. In the past months I've updated Citation.js several times, and the changes included a Node.js program for the commandline and better Wikidata input parsing. While I was working with the "old" code, I noticed some annoying issues in it.  One of the biggest things was the internal data format.

Citation.js Version 0.2: NPM package, CSL and more

Last week I made Citation.js usable for the command line, with Node.js. First of all, the main file determines if it is being run with Node.js or in the browser, and switches to Browser or Nodejs mode based on that.


Citation.js on the Command Line

Below is part two of a small series on ctj rdf, a new program I made to transform ContentMine CProjects into SPARQL-queryable Wikidata-linked rdf. Here’s a more detailed example. We are using my dataset available at 10.5281/zenodo.845935. It was generated from 1000 articles that mention ‘Pinus’ somewhere. This one has 15326 statements, whereof 8875 (57.9%) can be mapped, taking ~50 seconds.

ctj rdf: Relations Between Conifers Mentioned in Articles

Finally, Citation.js supports DOIs. It took a while, but it’s finally there. One big ‘but’: synchronous fetching doesn’t work in Chrome. I’m still looking into that, but I should be recommending you to use Cite.async() anyway. Also in this blog post: more stability in Cite#get(), a welcome byproduct of using the DOI API, and looking forward (again).   DOI support   So, DOIs. That was (and is) a though one. Let me guide you through the process.

Citation.js: DOI update and more stability

Below is part one of a small series on ctj rdf, a new program I made to transform ContentMine CProjects into SPARQL-queryable Wikidata-linked rdf. ctj has been around for longer, and started as a way to learn my way into the ContentMine pipeline of tools, but turned out to uncover a lot of possibilities in further processing the output of this pipeline (1, 2).  The recent addition of ctj rdf expands on this.

ctj rdf: Part One

The biblatex Package

On the Fediverse, @petrichor@digipres.club posited the question how to include identifiers for authors in Bib(La)TeX-based bibliographies:  I have wanted to try including ORCIDs in bibliographies for a while now, and while CSL-JSON makes it nearly trivial to encode, neither CSL styles nor CSL processors are at the point where those can actually be inserted in the formatted bibliography.

Including ORCID identifiers in BibLaTeX (and using them)

This past year was relatively quiet for Citation.js, as changes to the more prominent components (BibTeX, RIS, Wikidata) start to slow down. I believe this is a good sign, and that it indicates the quality of the mappings is high. Nonetheless, following the reworks of BibTeX and RIS in the past couple of years, some room for improvement still came up.

 Tytthaspis sedecimpunctata

, observed May 9th, 2021, Sint-Oedenrode, The Netherlands.

Citation.js: 2023 in review

Recently, I tried to find out the exact taxonomy of conifers. I knew that a few years earlier, when I was actively working with it, there were a few issues on Wikipedia concerning the grouping of the main conifer families, namely Araucariaceae, Cephalotaxaceae, Cupressaceae, Pinaceae, Podocarpaceae, Sciadopityaceae, Taxaceae, and actually the grouping of genera in families as well. Guess what changed: not much, not on Wikipedia anyway.


Conifer taxonomy

I worked on updates for Citation.js in the past few weeks, and I thought I'd go over them in this post. Async   First of all, Citation.js now has support for asynchronous parsing, so it won't lag your app as much when it uses e.g. the Wikidata API. This is good, as synchronous requests are not only blocking your app, but also deprecated in most major browsers.

Citation.js: Async, Showdown and Bib.TXT

Start of the project. What I am going to do: update Citation.js to accept quickscrape's results.json look into quickscrape and the other programs, mainly to see how I can make a similar command for my own project perhaps add/improve some scrapers try to make a card prototype for the database No real work yet, but some orientation and preparing until I have some more information.


Weekly Report 1: Start of my ContentMine fellowship

I updated Citation.js to accept quickscrape's results.json. Results can be viewed here. I made a card prototype for the database HTML page, using data extracted with quickscrape. It's mainly, although not advanced, a design thing, but it's nice to see what information I can fit. Results are here. With opened paper Without opened paper I have taken a look at quickscrape.


Weekly Report 2: Final project prototype

The main feature of the Library of Identification Resources is the description of the identification key (or matrix, reference, etc.). This description should on its basis specify when the key can or should be used.

On the modelling and application of the taxonomic coverage of identification keys

Almost two years ago I started participating on the then-new (now-archived) npm forum. I had been using npm for a few years at that point, and I had some free time to spend providing technical support, for fun. I fixed a number of bugs in the CLI, and users thanked me for those. My impact was limited, but the work was fulfilling. That is, until the developers at npm I had been working with got laid off.

Economics of open source versus open science

I made a test JavaScript file for NodeJS to practice working with it, and prepare more for the project.


Weekly Report 3: Playing with NodeJS

Version 0.5.0 Version 0.5.0 of Citation.js was released on April 1st, 2021. BibTeX and BibLaTeX After the update to the Bib(La)TeX file parser, described in the earlier

 BibTeX Rework: Syntax Update

blog post, the mapping of BibTeX and BibLaTeX data to CSL-JSON was also updated. The mapping is now split in two, one for BibLaTeX (which is backwards-compatible with BibTeX) and one for BibTeX.

Citation.js Version 0.5 and a 2022 update

Continuation of this post. I got an answer quite quickly (but after posting the previous post): @larswillighagen @Wikipedia try The Plant List. That's widely accepted: https://t.co/5J0AKPsCzD— R⓪ss Mounce (@rmounce) 20 augustus 2016  The Plant List marks what species are in what genus and family, and groups families in Major Groups, e.g. gymnosperms. It also marks synonyms.


Conifer taxonomy - Part 2

The "small program" proved more of a challenge than it seemed. Making a program to generate the JSON (link) was fairly easy. Loop through directories, find files, loop through files, collect XML data, save all collected data as JSON in a file. It took a while, but I think I spent the most time of it setting up the logistics, i.e. a nice logger, a file system reader and an argument processor.


Weekly Report 6: ContentMine output to JSON to HTML

So, I am a ContentMine fellow now. Actually, for a few weeks already. That's why you see the weekly reports. Those are reports about the project I am doing for my fellowship. I don not have much else to say about it. For more information, please take a look at the links below.


ContentMine Fellowship

(yes, three days late)

This week I wanted to catch up on all the things that had happened while I was on holiday. I have finished my introductory blogpost on ContentMine's blog, and I made this blog and transferred all the weekly reports from the GitHub Wiki to here. Next week will be more interesting.

Weekly Report 5: This is what pine literature looks like

This week I wanted to extend my program, with the lists of cards containing information. In the past few weeks, I made examples with topics such as conifers and zika. In the previous report I explained how I got the facts, and how other people could get them as well. But I felt it was a bit too complex, and mainly too messy.


Weekly Report 9: More topics

Last week I wanted to look into extracting more facts, and the relation between found species and compounds. This would be done by extending ami. However, it became clear there will be big improvements to ami in the future, and things like ChemicalTagger and OSCAR are planned to be implemented anyway. It's better to wait for those things to complete before extending it for my own purposes. Instead I improved the card page for future use.


Weekly Report 8: Visualising Zika articles

I changed the "JSON maker" I used to convert CProjects to JSON last week to be useful for a more general use case, being more user-friendly and having more options, like grouping by species and minifying. It's now called

 ctj

(CProject to JSON), although that name may be changed to something more clear or appropriate. The GitHub repository can be found here. ctj CProjects are the output of the ContentMine tools.

CProject to JSON

Today I updated my GitHub repository of Citation.js. The main difference in the source files was the correction of a typo, but I updated the docs and restructured the directories as well.


Citation.js

I was on holiday for the last two weeks, so I was not able to do all to much and I missed the second webinar, but I did read some on-topic papers about NLP.


Weekly Report 4: Learning about text mining

Extended proposal ( 3 June 2016, GitHub, with comments ): Analyzer and data visualizer for papers about conifers


 Description

A program visualizing and listing data extracted with ContentMine software from a list of papers resulting from a query. Different types of data (authors, genes, molecules etc.) will be listed in different columns.

Research project proposals

Syntaxus baccata

The [CDLI](https://cdli.ucla.edu/) is developing a [new
website](https://cdli-dev.mpiwg-berlin.mpg.de/). That website's admin
interface for its journals contains a page where a LaTeX source file,
following a specific template, is configured to an HTML page. For this,
apart from the LaTeX file itself, two additional components are needed:
a BibTeX file, containing metadata of the references; and image files.

### Current implementation

The current implementation involves uploading images separately from the
form that creates the article. This can lead to "orphan" images if such
a form is abandoned after uploading the images. The current
implementation has an additional problem, where files are saved in a
single directory, so multiple images with the same file name (say,
`Figure1.jpg`) will overwrite previous images.

``` language-mermaid
#mermaid-svg-zuMp4V2cXsJ0PnCG{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#000000;}#mermaid-svg-zuMp4V2cXsJ0PnCG .error-icon{fill:#552222;}#mermaid-svg-zuMp4V2cXsJ0PnCG .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-zuMp4V2cXsJ0PnCG .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-zuMp4V2cXsJ0PnCG .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-zuMp4V2cXsJ0PnCG .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-zuMp4V2cXsJ0PnCG .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-zuMp4V2cXsJ0PnCG .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-zuMp4V2cXsJ0PnCG .marker{fill:#666;stroke:#666;}#mermaid-svg-zuMp4V2cXsJ0PnCG .marker.cross{stroke:#666;}#mermaid-svg-zuMp4V2cXsJ0PnCG svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-zuMp4V2cXsJ0PnCG .actor{stroke:hsl(0,0%,83%);fill:#eee;}#mermaid-svg-zuMp4V2cXsJ0PnCG text.actor > tspan{fill:#333;stroke:none;}#mermaid-svg-zuMp4V2cXsJ0PnCG .actor-line{stroke:#666;}#mermaid-svg-zuMp4V2cXsJ0PnCG .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-zuMp4V2cXsJ0PnCG .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-zuMp4V2cXsJ0PnCG #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-zuMp4V2cXsJ0PnCG .sequenceNumber{fill:white;}#mermaid-svg-zuMp4V2cXsJ0PnCG #sequencenumber{fill:#333;}#mermaid-svg-zuMp4V2cXsJ0PnCG #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-zuMp4V2cXsJ0PnCG .messageText{fill:#333;stroke:#333;}#mermaid-svg-zuMp4V2cXsJ0PnCG .labelBox{stroke:hsl(0,0%,83%);fill:#eee;}#mermaid-svg-zuMp4V2cXsJ0PnCG .labelText,#mermaid-svg-zuMp4V2cXsJ0PnCG .labelText > tspan{fill:#333;stroke:none;}#mermaid-svg-zuMp4V2cXsJ0PnCG .loopText,#mermaid-svg-zuMp4V2cXsJ0PnCG .loopText > tspan{fill:#333;stroke:none;}#mermaid-svg-zuMp4V2cXsJ0PnCG .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(0,0%,83%);fill:hsl(0,0%,83%);}#mermaid-svg-zuMp4V2cXsJ0PnCG .note{stroke:hsl(60,100%,23.3333333333%);fill:#ffa;}#mermaid-svg-zuMp4V2cXsJ0PnCG .noteText,#mermaid-svg-zuMp4V2cXsJ0PnCG .noteText > tspan{fill:#333;stroke:none;}#mermaid-svg-zuMp4V2cXsJ0PnCG .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-zuMp4V2cXsJ0PnCG .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-zuMp4V2cXsJ0PnCG .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-zuMp4V2cXsJ0PnCG:root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-zuMp4V2cXsJ0PnCG sequence{fill:apa;}ClientServerGET /admin/articles/add/cdlj200 (add page)POST /admin/articles/convert-latex200 (converted HTML, list of images)POST /admin/articles/image 'Figure1.jpg'Saves image in 'Figure1.jpg'200POST /admin/articles/add/cdlj300 /articles/ClientServer
```

### New implementation

With a new implementation of the rest of the forms, simplifying a lot of
the code, the issues with the image uploads are also attempted to be
resolved.

#### Saving images together with the metadata

To avoid the problem of the "orphan" images resulting from abandoned
forms, one could submit the images in the same form that creates the
article. If that form is not submitted, or contains invalid data so
cannot be saved, the images will not be uploaded.

#### Saving images according to metadata

The second problem could be solved immediately by saving the images in
subdirectories according to the metadata of the image, e.g. `2021-01/`
where `2021` is the year the article is published and `01` is the
article sequence number within that year. This however assumes that that
metadata does not change after the initial submission.

Both these solutions however creates some constraints on other problems,
because it means that the images can only be saved *after* the user
submits the main form, so *after* the HTML containing

elements with references to the image locations is generated. Somehow,
those image locations should be able to identify the article, *before*
any information about that article is known:

``` language-mermaid
#mermaid-svg-6CZ5geXR2BJYqtFv{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#000000;}#mermaid-svg-6CZ5geXR2BJYqtFv .error-icon{fill:#552222;}#mermaid-svg-6CZ5geXR2BJYqtFv .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-6CZ5geXR2BJYqtFv .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-6CZ5geXR2BJYqtFv .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-6CZ5geXR2BJYqtFv .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-6CZ5geXR2BJYqtFv .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-6CZ5geXR2BJYqtFv .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-6CZ5geXR2BJYqtFv .marker{fill:#666;stroke:#666;}#mermaid-svg-6CZ5geXR2BJYqtFv .marker.cross{stroke:#666;}#mermaid-svg-6CZ5geXR2BJYqtFv svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-6CZ5geXR2BJYqtFv .actor{stroke:hsl(0,0%,83%);fill:#eee;}#mermaid-svg-6CZ5geXR2BJYqtFv text.actor > tspan{fill:#333;stroke:none;}#mermaid-svg-6CZ5geXR2BJYqtFv .actor-line{stroke:#666;}#mermaid-svg-6CZ5geXR2BJYqtFv .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-6CZ5geXR2BJYqtFv .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-6CZ5geXR2BJYqtFv #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-6CZ5geXR2BJYqtFv .sequenceNumber{fill:white;}#mermaid-svg-6CZ5geXR2BJYqtFv #sequencenumber{fill:#333;}#mermaid-svg-6CZ5geXR2BJYqtFv #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-6CZ5geXR2BJYqtFv .messageText{fill:#333;stroke:#333;}#mermaid-svg-6CZ5geXR2BJYqtFv .labelBox{stroke:hsl(0,0%,83%);fill:#eee;}#mermaid-svg-6CZ5geXR2BJYqtFv .labelText,#mermaid-svg-6CZ5geXR2BJYqtFv .labelText > tspan{fill:#333;stroke:none;}#mermaid-svg-6CZ5geXR2BJYqtFv .loopText,#mermaid-svg-6CZ5geXR2BJYqtFv .loopText > tspan{fill:#333;stroke:none;}#mermaid-svg-6CZ5geXR2BJYqtFv .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(0,0%,83%);fill:hsl(0,0%,83%);}#mermaid-svg-6CZ5geXR2BJYqtFv .note{stroke:hsl(60,100%,23.3333333333%);fill:#ffa;}#mermaid-svg-6CZ5geXR2BJYqtFv .noteText,#mermaid-svg-6CZ5geXR2BJYqtFv .noteText > tspan{fill:#333;stroke:none;}#mermaid-svg-6CZ5geXR2BJYqtFv .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-6CZ5geXR2BJYqtFv .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-6CZ5geXR2BJYqtFv .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-6CZ5geXR2BJYqtFv:root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-6CZ5geXR2BJYqtFv sequence{fill:apa;}ClientServerGET /admin/articles/add/cdlj200 (add page)POST /admin/articles/convert-latexAt this point the HTMLshould contain linksto the permanent locations of the images200 (converted HTML, list of images)POST /admin/articles/add/cdljAt this pointthe images are savedin a permanent location300 /articles/ClientServer
```

So, what is the solution? I propose the following:

``` language-mermaid
#mermaid-svg-OR3KFJWzCHNmFnJn{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#000000;}#mermaid-svg-OR3KFJWzCHNmFnJn .error-icon{fill:#552222;}#mermaid-svg-OR3KFJWzCHNmFnJn .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-OR3KFJWzCHNmFnJn .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-OR3KFJWzCHNmFnJn .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-OR3KFJWzCHNmFnJn .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-OR3KFJWzCHNmFnJn .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-OR3KFJWzCHNmFnJn .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-OR3KFJWzCHNmFnJn .marker{fill:#666;stroke:#666;}#mermaid-svg-OR3KFJWzCHNmFnJn .marker.cross{stroke:#666;}#mermaid-svg-OR3KFJWzCHNmFnJn svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-OR3KFJWzCHNmFnJn .actor{stroke:hsl(0,0%,83%);fill:#eee;}#mermaid-svg-OR3KFJWzCHNmFnJn text.actor > tspan{fill:#333;stroke:none;}#mermaid-svg-OR3KFJWzCHNmFnJn .actor-line{stroke:#666;}#mermaid-svg-OR3KFJWzCHNmFnJn .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-OR3KFJWzCHNmFnJn .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-OR3KFJWzCHNmFnJn #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-OR3KFJWzCHNmFnJn .sequenceNumber{fill:white;}#mermaid-svg-OR3KFJWzCHNmFnJn #sequencenumber{fill:#333;}#mermaid-svg-OR3KFJWzCHNmFnJn #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-OR3KFJWzCHNmFnJn .messageText{fill:#333;stroke:#333;}#mermaid-svg-OR3KFJWzCHNmFnJn .labelBox{stroke:hsl(0,0%,83%);fill:#eee;}#mermaid-svg-OR3KFJWzCHNmFnJn .labelText,#mermaid-svg-OR3KFJWzCHNmFnJn .labelText > tspan{fill:#333;stroke:none;}#mermaid-svg-OR3KFJWzCHNmFnJn .loopText,#mermaid-svg-OR3KFJWzCHNmFnJn .loopText > tspan{fill:#333;stroke:none;}#mermaid-svg-OR3KFJWzCHNmFnJn .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(0,0%,83%);fill:hsl(0,0%,83%);}#mermaid-svg-OR3KFJWzCHNmFnJn .note{stroke:hsl(60,100%,23.3333333333%);fill:#ffa;}#mermaid-svg-OR3KFJWzCHNmFnJn .noteText,#mermaid-svg-OR3KFJWzCHNmFnJn .noteText > tspan{fill:#333;stroke:none;}#mermaid-svg-OR3KFJWzCHNmFnJn .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-OR3KFJWzCHNmFnJn .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-OR3KFJWzCHNmFnJn .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-OR3KFJWzCHNmFnJn:root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-OR3KFJWzCHNmFnJn sequence{fill:apa;}ClientServerGET /admin/articles/add/cdljGenerate random article ID200 (add page with embedded article id)POST /admin/articles/convert-latexGenerate image URLSaccording torandom article ID200 (converted HTML, list of images)POST /admin/articles/add/cdljVerify random article ID,save images,generate sequential ID300 /articles/ClientServer
```


Since [Google Summer of Code](https://summerofcode.withgoogle.com/) 2020
I have been contributing code to the new framework of the [Cuneiform
Digital Library Initiative](https://cdli.mpiwg-berlin.mpg.de/) (CDLI), a
digital repository for metadata, transliterations, images, and other
data of cuneiform inscriptions, as well as tools to work with that data.

One of the features of the new CDLI framework is improved editing of
artifact metadata, as well as inscriptions. There are several ways
artifacts will be able to be edited: by uploading metadata in a
CSV-format, by batch edits in the website interface, and by editing
individual artifacts. At the basis of all those pathways is the database
representation of individual revisions. To evaluate the planned
representation, and to see if alternatives are possible, I took a look
at the catalogue metadata (CDLI 2021), and what edits are being made
currently.

### Artifact registrations

![](https://1.bp.blogspot.com/-Wmmc4zWTVu0/YGJSQNJ_-SI/AAAAAAAAuyE/pwudg-SVFaI4FAQqlEbJsBMuVVcY4vYxACLcBGAsYHQ/s0/collection_year2.png) 
***Figure 1: The number of artifacts registered each year, split by the
collection they are in.** Collections with less than 7000 artifacts
represented are combined into the "Other" category to keep the legend
manageable. 13K artifacts without valid creation dates were excluded.*

First, I took a quick look at the composition of the catalogue (Fig. 1).
As it turns out some collections had most of their (included) artifacts
added in a single year, such as the University of Pennsylvania Museum of
Archaeology and Anthropology in 2005 and the Anadolu Medeniyetleri
Müzesi in 2012. Other collections seemed to have had a more steady flows
of artifact entries, particularly the British Museum. Overall, this does
not help much with selecting a database representation of revisions
though.

One of the options we want to evaluate is storing revisions in a linked
format, similarly to how artifacts are stored now, instead of a flat
format. This means that if each of those artifacts has about 10 links in
total --- say 1 material, 1 genre, 1 holding collection, 1 language, 1
associated composite, and 5 publications --- each revision would need 10
rows for the links and 1 row for the rest of the entry. Therefore, the
question is: are 11 rows per revision manageable for 343,371 active
artifacts?

### Revisions

To find out, let's take a look at the daily updates of the catalogue
data. With the git history, we can find out how many artifacts were
edited on each day. Since the commits are made daily multiple
consecutive edits to the same artifact are counted as a single revision.
On the other hand, the removal of an artifact from the catalogue might
be counted as a revision. Whether that balances out is hard to tell, so
these numbers are a rough estimate. The analysis only goes back to 2017
unfortunately, as before that the catalogue was included as a ZIP file.

![](https://1.bp.blogspot.com/-cUbyYip36v4/YGJSR2ar-iI/AAAAAAAAuyI/kP-JfpOVGhMGQ3G0ETf3xaM9VC9fNvRpACLcBGAsYHQ/s0/figure.png) 
***Figure 2: Number of artifact revisions per year.** The 12 largest
revisions are highlighted and explained below.*

Figure 2 highlights in various colors the 12 revisions affecting the
highest number of artifacts. Most of these are 7 consecutive in October
and November of 2017. These involved editing the ID column, something
which should not happen in the current system. Other large revision
usually affected a single column, thereby revising almost every
artifact:

- Padding designation numbers with leading zeroes ("CDLI Lexical 000030,
 ex. 13" → "013"):
 [2020-02-11](https://github.com/cdli-gh/data/commit/8bb9f40470e3d9650b40720e86ba2884bb37667f),
 [2018-01-26](https://github.com/cdli-gh/data/commit/b469b6167b12b8fc266b6ebcd699dc86185cfa8d)
- Addition of new columns:
 [2019-09-10](https://github.com/cdli-gh/data/commit/84aa2655bfd70b0c2cce868cee47c47e697cfb84)
- New default values (\"\" → "no translation" for the column
 `translation_source`):
 [2018-10-30](https://github.com/cdli-gh/data/commit/3cf96f26440ea99a400582bb25925e59f41f8b81)

Outside the top 12 the edits become a lot more meaningful. Often,
credits for new transliterations or translations are added, sometimes
with the previously undetermined language now specified.

As it turns out, approximately 3 million edits are made every year. If
all those edits are stored as linked entities, we are looking at 30
million table rows, *per year*. However, even if the edits are stored in
a flat format there would be 3 million table rows per year already.
Either option might become a problem in 5--10 years, depending on SQL
performance. With indexing it might not be a problem at all: usually the
only query is by identifier anyway.

### Changed fields

That said, let's presume I choose the flat format. Most revisions only
change one or two fields (excluding the modification date which would
not be included). Duplicating the whole row might be wasteful, so what
could I do to avoid that?

Since the flat format is based on the schema of the CSV files used for
batch edits, each column can be considered as text, with an empty string
for empty values. This leaves `NULL` (i.e. "no value") available to
represent "no change". Together with MySQL's `SPARSE` columns only
edited fields would be stored. (Otherwise, each empty value would need
to be signified as such. Now, actual values carry some extra information
to the same end.)

It would also make it even easier to display a list of changes, as there
is no need to compare the value with the previous one. Other operations
with revisions, such as merging two revisions made simultaneously on the
same version of an artifact, would be easier for the same reason.

Since this would not be possible, or not as easy, with a linked format
perhaps it was good the shear volume of edits pointed me that way
anyway.

### References

Cuneiform Digital Library Initiative. (2021, March 8). Monthly release
2021.03 (Version 2021.03). Zenodo.
<http://doi.org/10.5281/zenodo.4588551>