Maastricht University

Egon Willighagen

[Scholia](https://scholia.toolforge.org/) is a visual layer on top of
[Wikidata](https://wikidata.org/) providing a rich user experience for
browing scholarly research related knowledge. I am using the combinatie
for various things, including exploring new research topics (a method,
compound, or protein I do not know so much about yet), indexing notable
research output (including citations), [progress of Citation Typing
Ontology uptake](https://chem-bla-ics.linkedchemistry.info/tag/cito),
etc. This weekend I hope to send around the final draft for the *Scholia
Chemistry* paper.

Scholia has received a fair share of scholarly and social attention. The
Scholia paper has been cited [over 100
times](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=scholia+wikidata&btnG=&oq=scholia)
and the websites received about 200 thousand page views each day (though
we do not know how to get Toolforge to give us sufficient insight into
the how and what of that count). There is a Wikipedia template to link
to Scholia and some of projects I am involved in link Scholia for
articles, such as [WikiPathways](https://wikipathways.org/).

With that, there is also interest in using it for other Wikibases and
perhaps even random SPARQL endpoints. These things are not trivial, as
Scholia uses complementary APIs, various URL patterns for some of the
functionality, and generally, all SPARQL queries are tweaked to the
Wikidata Blazegraph SPARQL endpoint to ensure results are returned in
reasonable time. But that last requires use of Blazegraph extensions to
the SPARQL standard.

All this requires Scholia to become more independent, in a better
model-view-controller model. And that actually turns out very important
at this moment. That is, Wikidata is not a RDF-first database, but a
Wikibase-based store. Whenever an edit is made, RDF is generated and the
SPARQL endpoint is updated. Now, the number of edits in Wikidata is
enormous and the notion that the SPARQL endpoint is often minutes at
most behind is a huge accomplishment. But the Blazegraph platform cannot
keep up with Wikidata. Blazegraph is open source, but has been bought up
and development stopped from one day to another.

Therefore, a split of the Wikidata SPARQL platform is
[planned](https://phabricator.wikimedia.org/T337013). This split will
put one part of the knowledge in on endpoint and the other half in the
other. Any query that needs information from both graphs, will have to
do a federated SPARQL query. Basically, there are very few Scholia
queries that do not rewriting. My first rewrite actually failed, because
the rewriting is not obvious and quickly times out. To some extend, this
is because now lots of results of subqueries need to be send over the
network from one endpoint to the other. When the combined query
basically covers half of each endpoint, that's a lot of network traffic.

An immediate use case of the configuration is therefore running Scholia
against the current three endpoints: the current official endpoint, and
the two split endpoints under development. With [a recent
patch](https://github.com/WDscholia/scholia/pull/2515)
[Finn](@fnielsen@expressional.social) and I worked on, this
configuration looks like this (and saved as
`scholia.ini`{.language-plaintext .highlighter-rouge}:

:::: {.language-plaintext .highlighter-rouge}
::: {.highlight}
``` highlight
[query-server]
# Wikidata:
#sparql_endpoint = https://query.wikidata.org/sparql
#sparql_editurl = https://query.wikidata.org/#
#sparql_embedurl = https://query.wikidata.org/embed.html#

# Wikidata Split Main
sparql_endpoint = https://query-main-experimental.wikidata.org/sparql
sparql_editurl = https://query-main-experimental.wikidata.org/#
sparql_embedurl = https://query-main-experimental.wikidata.org/embed.html#

# Wikidata Split Scholar
#sparql_endpoint = https://query-scholarly-experimental.wikidata.org/sparql
#sparql_editurl = https://query-scholarly-experimental.wikidata.org/#
#sparql_embedurl = https://query-scholarly-experimental.wikidata.org/embed.html#
```
:::
::::

So, right now, we can test the impact of the split with Scholia and this
patch. We would fire up a local instances of Scholia, running against
one of the split endpoints, and use the Toolforge instance as baseline.

Now, on my system I need to use [Python
virtualenv](https://python.land/virtual-environments/virtualenv) so, I
first start a Scholia `venv`{.language-plaintext .highlighter-rouge}:

:::: {.language-shell .highlighter-rouge}
::: {.highlight}
``` highlight
source ~/.venvs/scholia/bin/activate
```
:::
::::

After that, I can select an other endpoint, e.g. the
`main`{.language-plaintext .highlighter-rouge} Wikidata split endpoint
(`query-main-experimental.wikidata.org`{.language-plaintext
.highlighter-rouge}) were it not they are [currently
offline](https://phabricator.wikimedia.org/T371833) as part of the
transition and run Scholia on a unique port:

:::: {.language-shell .highlighter-rouge}
::: {.highlight}
``` highlight
scholia run
```
:::
::::

Then I can have two browser windows along side and compare Scholia pages
againt the current Scholia instance and when running against another
SPARQL endpoint. For now, I can test how well Scholia runs on the
[QLever instance of Wikidata](qlever.cs.uni-freiburg.de/wikidata)
(superfast and updated data once a week). Here the configuration I have
is not entirely complete, and many SPARQL queries do not work against
QLever, including anything with graphical depiction. But that said, I
can use this configuration:

:::: {.language-plaintext .highlighter-rouge}
::: {.highlight}
``` highlight
[query-server]
# QLever
#sparql_endpoint = https://qlever.cs.uni-freiburg.de/api/wikidata
#sparql_editurl = https://qlever.cs.uni-freiburg.de/wikidata/?query=
#sparql_embedurl = 
```
:::
::::

Then, I can compare, for example, the chemicals statistics the main
Scholia with one running against QLever:

![](/assets/images/scholia_comparison.png)

This query ran without modification. For other queries rewriting is
needed, but with this setup we can at least quickly see the differences
in the results.

#### References

- [10.1007/978-3-319-70407-4_36](https://doi.org/10.1007/978-3-319-70407-4_36)


Scholia is a visual layer on top of Wikidata providing a rich user experience for browing scholarly research related knowledge. I am using the combinatie for various things, including exploring new research topics (a method, compound, or protein I do not know so much about yet), indexing notable research output (including citations), progress of Citation Typing Ontology uptake, etc.

Scholia configurability

Just before the end of the year, the *Wikidata subsetting: approaches,
tools, and evaluation* paper by Seyed Amir Hosseini Beghaeiraveri *et
al.* got published
(doi:[10.3233/SW-233491](https://doi.org/10.3233/SW-233491)). I am
really excited our group (i.e.
[Ammar](https://orcid.org/0000-0002-8399-8990) and
[Denise](https://orcid.org/0000-0001-8449-1318)) has been able to
contribute to this. I think it also is a great example of the power of
hackathons to bring together people.

To me, subsetting of Wikidata (or any large knowledge graph) is
important for a couple of reasons. First, there can be practical
reasons. Scholia, for example, is computationally expensive, and the
idea we explore in the Alfred P. Sloan Foundation grant for Scholia
(doi:[10.3897/rio.5.e35820](https://doi.org/10.3897/rio.5.e35820)) was
that a subset of Wikidata would make it more performant and potentially
more environmental-friendly.

A second reason is more about the scientific process. When doing an
analysis and when you want to make the reasoning transparent, you want
to share the analyzed data as part of the research output (basically,
the "data"). For example, the data may have undergone some curation, or
you combined data from two or more different sources. And you will want
to share this as part of the scientific process. Resharing a full dump
of the larger knowledge base would not be practical for at least two
reasons: duplication of huge data, and a lot of unrelated content makes
it hard for peers to find the bits of interest to the study.

Subsetting may be useful here. This paper evaluates a number of
different subsetting approaches. Myself, I am particularly excited about
the idea that we can take a shape expression (e.g.
[ShEx](https://shex.io)) as input. I still love the idea that I take the
SPARQL queries in my analyses, convert that into shapes automatically,
and then get a subet that returns the exact same results as the query
would on the full dataset.


Just before the end of the year, the
<em>
 Wikidata subsetting: approaches, tools, and evaluation
</em>
paper by Seyed Amir Hosseini Beghaeiraveri
<em>
 et al.
</em>
got published (doi:10.3233/SW-233491). I am really excited our group (i.e. Ammar and Denise) has been able to contribute to this. I think it also is a great example of the power of hackathons to bring together people.

New paper: “Wikidata subsetting: approaches, tools, and evaluation”

About four weeks ago the [Fall 2023 American Chemical
Society](https://www.acs.org/meetings/acs-meetings/fall-2023.html)
meeting ([\#ACSFall2023](https://mastodon.social/tags/ACSFall2023)). I
have attended a few ACS meetings in person and even organized a
[symposium at the 2010 ACS meeting](https://egonw.github.io/acsrdf2010/)
in Boston. This time too, I did not participate in person, tho visiting
San Francisco again would have been nice. I gave
[two](https://mastodon.social/@egonw@social.edu.nl/110882509829434765)
[presentations](https://mastodon.social/@egonw@social.edu.nl/110883271752255923)
(slides
doi:[10.5281/zenodo.8255394](https://doi.org/10.5281/zenodo.8255394)),
but have not uploaded my slides of the first presentation to Zenodo yet.

The theme of the meeting was data, and this resulted in a wealth of
presentations with cheminformatics. What is striking here is that a lot
of work has not changed so much in 20 years, except for the scale. What
I missed here was the large open data sets, but generally the level of
open science was heartwarming! So many preprints mentions, GitHub
repositories, and Zenodo deposits. The Blue Obelisk was truly ahead of
its time, but it is a delight to see the field of chemistry catch up. I
can now say a lot of about peer review, and why the field is not
benefitting from all the experience that exists in the field because
people publish in the wrong journals, but that is for another time.

I attended multiple sessions, which is a bit of a challenge, doing this
remotely from Central European Summer Time (CEST). Of course, the Sunday
started with the [Chemical informatics (R)evolution: Towards
Democratization and Open
Science](https://acs.digitellinc.com/sessions/574129/view) session,
where I had my first talk, and later that day the [Enhance your Data -
Smart Ways to Metadata and Knowledge
Graphs](https://acs.digitellinc.com/sessions/573932/view) session, where
I gave a second talk, about [Bioschemas](https://bioschemas.org/)'
`ChemicalSubstance`{.language-plaintext .highlighter-rouge} and
`MolecularEntity`{.language-plaintext .highlighter-rouge}. Sadly, I had
to leave that meeting early because it was getting too late.

There were so many interesting sessions, I could not attend everything.
I also have to go back to all [my
notes](https://mastodon.social/tags/ACSFall2023) and isolate things I
want to follow up on, prominently open datasets.

More later.


About four weeks ago the Fall 2023 American Chemical Society meeting (#ACSFall2023). I have attended a few ACS meetings in person and even organized a symposium at the 2010 ACS meeting in Boston. This time too, I did not participate in person, tho visiting San Francisco again would have been nice. I gave two presentations (slides doi:10.5281/zenodo.8255394), but have not uploaded my slides of the first presentation to Zenodo yet.

Messaggi di Rogue Scholar

Scholia configurability

New paper: “Wikidata subsetting: approaches, tools, and evaluation”

American Chemical Society Fall 2023 meeting