Every year, the Commandant of the Marine Corps publishes a reading list of books that often only bear on warfighting tangentially at best. The idea behind this is that those entrusted with the lives of servicemembers should have an understanding of the world that goes beyond the profession of arms. In much the same way, I have been advising data scientists to go beyond professional literature.

fulltext is a package I maintain for text-mining the scholarly literature (package docs). You can search for articles, fetch article metadata and abstracts, and fetch full text of some articles. Text-mining the scholarly literature is a research tool used across disciplines. Full text of articles (entire article, not just the abstract) is the gold standard in text-mining in most cases.

pubchunks is a package grown out of the fulltext package. fulltextprovides a single interface to many sources of full text scholarly articles. Aspart of the user flow in fulltext there is an extraction step where fulltext::chunks()pulls parts of articles out of XML format article files.

As a regular user of the scholarly literature since before the internet (I started reading primary scientific literature for a high-school project around 1989), I have closely followed its digitization.

This morning I was reminded of the age of some of the technology we’re using. Hyperlinks were developed at Stanford University and first demonstrated by their inventor Douglas Engelbart (using the first mouse) in 1968: On Tuesday, Douglas Engelbart died, even before the scholarly literature was able to fully implement the technology he invented, 45 years and counting.

Tomorrow is the Anchoring Biodiversity Information: From Sherborn to the 21st century and beyond meeting.