Jeroen Ooms

This week an update for
[xml2](https://cran.r-project.org/web/packages/xml2/index.html) and a
new [xslt](https://cran.r-project.org/web/packages/xslt/index.html)
package have appeared on CRAN. A full announcement for xml2 version 1.1
will appear on the [rstudio blog](https://blog.rstudio.org/). This post
explains xml *validation* (via xsd schema) and xml *transformation* (via
xslt stylesheets) which have been added in this release.

XML schemas and stylesheets are not exactly new; both [xslt
1.1](https://www.w3.org/TR/xslt11/) (2001) and [xsd
1.0](https://www.w3.org/TR/xmlschema-1/) (2004) have been available in
browsers for over a decade. Revised specifications for xsd/xslt are
still developed, but not widely implemented due to declined popularity
of xml itself. Our R implementation builds on
[libxslt](http://xmlsoft.org/libxslt/) which supports XSLT 1.0 features
plus most of the EXSLT set of processor-portable extensions functions.

## XML Validation with XSD

XML schema, also referred to as XSD (XML Schema Definition) is standard
for defining the fields and formats that are supposed to appear within
an XML document. This provides a formal method for validating XML
messages. The schema itself is also written in XML (there is even an
[xsd schema](https://www.w3.org/2001/XMLSchema.xsd) for validating xml
schemas).

This
[example](https://msdn.microsoft.com/en-us/library/ms256129(v=vs.110).aspx)
from msdn illustrates the idea using a schema for a hypothetical
purchase order. Imagine a vendor has an XML api for retailers to
automatically order products. The order can be quite complex but the
schema formally describes what constitutes a valid XML order message. It
contains fields like this:

::: {.highlight}
``` {.chroma tabindex="0"}
 <xs:complexType name="PurchaseOrderType">  <xs:sequence>   <xs:element name="shipTo" type="USAddress"/>   <xs:element name="billTo" type="USAddress"/>   <xs:element ref="comment" minOccurs="0"/>   <xs:element name="items"  type="Items"/>  </xs:sequence>  <xs:attribute name="orderDate" type="xs:date"/> </xs:complexType>
```
:::

Both the client and server can easily validate an XML order against this
schema to ensure that all required fields are present and contain the
correct format. A copy of this example is included with the `xml2`
package:

::: {.highlight}
``` {.chroma tabindex="0"}
# Example orderdoc <- read_xml(system.file("extdata/order-doc.xml", package = "xml2"))# Example schemaschema <- read_xml(system.file("extdata/order-schema.xml", package = "xml2"))xml_validate(doc, schema)# TRUE
```
:::

The `xml_validate` function returns TRUE or FALSE. If FALSE it also
contains an attribute with a data frame listing invalid elements in the
XML document. Let's replace some text in the XML document to make it
invalid:

::: {.highlight}
``` {.chroma tabindex="0"}
# Create invalid order to teststr <- readLines(system.file("extdata/order-doc.xml", package = "xml2"))str <- sub("<quantity>1", "<quantity>", str)str <- sub("95819", "ABC95819", str)str <- sub('partNum="926-AA"', "", str)doc <- read_xml(paste(str, collapse = "\n"))
```
:::

This new document will fail validation. The return object from
`xml_validate` contains an `error` attribute with a dataframe containing
the validation errors.

::: {.highlight}
``` {.chroma tabindex="0"}
# Fails validationout <- xml_validate(doc, schema)# Show the errorsattr(out, "errors")#[1] "Element 'zip': 'ABC95819' is not a valid value of the atomic type 'xs:decimal'."#[2] "Element 'quantity': '' is not a valid value of the local atomic type."#[3] "Element 'item': The attribute 'partNum' is required but missing."#[4] "Element 'quantity': '' is not a valid value of the local atomic type."
```
:::

When implementing an R client for a system with an XML API which also
provides a schema, it is good practice to validate your messages before
submitting them to the server. Thereby you catch problems with your XML
document locally.

## XML Transformation with XSL

Extensible Stylesheet Language (XSL) Transformation provides a
standardized language for converting a certain XML structure into
another XML or HTML structure. Usually the original xml document
provides the raw data, and the stylesheet contains a template for a HTML
page that presents this content. Again, the XSLT document itself is also
written in XML.

We have decided to implement this in a separate package called `xslt`
because it requires another C library. Try the example from the
`xml_xslt` manual page:

::: {.highlight}
``` {.chroma tabindex="0"}
library(xslt)doc <- read_xml(system.file("examples/cd_catalog.xml", package = "xslt"))style <- read_xml(system.file("examples/cd_catalog.xsl", package = "xslt"))html <- xml_xslt(doc, style)cat(as.character(html))
```
:::

This example is explained in more detail on
[w3schools](https://www.w3schools.com/xml/xsl_transformation.asp).

## Why Use XSLT?

As the name implies, XSLT is designed to apply styling so that we can
separate data of a document from its presentation markup. Take this
[example](https://msdn.microsoft.com/nl-nl/library/ms765388(v=vs.85).aspx)
of an XSLT document from the msdn homepage:

::: {.highlight}
``` {.chroma tabindex="0"}
<?xml version="1.0"?><xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">  <xsl:template match="/hello-world">    <HTML>      <HEAD>        <TITLE></TITLE>      </HEAD>      <BODY>        <H1>          <xsl:value-of select="greeting"/>        </H1>        <xsl:apply-templates select="greeter"/>      </BODY>    </HTML>  </xsl:template>  <xsl:template match="greeter">    <DIV>from <I><xsl:value-of select="."/></I></DIV>  </xsl:template></xsl:stylesheet>
```
:::

Now if we apply this to a document like this:

::: {.highlight}
``` {.chroma tabindex="0"}
<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="hello.xsl"?><hello-world>   <greeter>An XSLT Programmer</greeter>   <greeting>Hello, World!</greeting></hello-world>
```
:::

We get the following output:

::: {.highlight}
``` {.chroma tabindex="0"}
<HTML><HEAD><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><TITLE></TITLE></HEAD><BODY><H1>Hello, World!</H1><DIV>from <I>An XSLT Programmer</I></DIV></BODY></HTML>
```
:::

When XSLT was introduced in [1999](https://www.w3.org/TR/xslt), it was
expected that xml would replace html. Computer scientists envisioned
that dynamic content of websites would be served via semantically
structured xmls feeds (such as RSS), and presentation markup (i.e. a
nice html page) could be added on the client by applying a fixed
transformation.

Unfortunately that's now how it went. It turned out that xslt was overly
complex and never really found wide adoption. Instead people started
writing dynamic HTML pages using PHP, which was slow and insecure, but
considerably easier to learn. And that brings us back to R :)


This week an update for xml2 and a new xslt package have appeared on CRAN. A full announcement for xml2 version 1.1 will appear on the rstudio blog. This post explains xml
<em>
 validation
</em>
(via xsd schema) and xml
<em>
 transformation
</em>
(via xslt stylesheets) which have been added in this release. XML schemas and stylesheets are not exactly new; both xslt 1.1 (2001) and xsd 1.0 (2004) have been available in browsers for over a decade.

Using xml schema and xslt in R

University of Glasgow

Roderic Page

One of the things I keep revisiting is the way we display scientific
articles. Apart from Nature\'s excellent
[iPhone](http://iphylo.blogspot.com/2010/08/viewing-scientific-articles-on-ipad.html)
and iPad apps, most efforts to re-imagine how we display articles are
little more than glorified PDF viewers (e.g., [the PLoS iPad
app](http://iphylo.blogspot.com/2010/08/viewing-scientific-articles-on-ipad_24.html)).  
  
Part of the challenge is that if we make the article more interactive we
immediately confront the problem of how to link to other content. For
example, we may have a lovingly crafted ePub view (e.g., Nature\'s
apps), but what happens when the user clicks on a citation to another
paper? If the paper is published by the same journal, then potentially
it could be viewed using the same viewer, but if not then we are at the
mercy of the other publisher. They will have their own ideas of how to
display articles, so the simplest fallback is to display the cited
article in a web browser view. The problem with this is that it breaks
the user experience - the other publisher is unlikely to follow the same
conventions for displaying an article and its links. If we are lucky the
cited article might be published in an Open Access journal that
provides, say, XML based on the NLM DTD standard. Knowing whether an
article is [Open Access or not is not
straightforward](http://iphylo.blogspot.com/2010/12/how-do-i-know-if-article-is-open-access.html),
and different journals have their own unique interpretation of the NLM
standard.  
  
Then there is the issue of other kinds of content, such as taxonomic
names, specimens, DNA sequences, geographic localities, etc. We lack
[decent
services](http://iphylo.blogspot.com/2010/04/time-for-some-decent-service.html)
for many of these objects, as a result efforts like [PLoS Biodiversity
Hub](http://hubs.plos.org/web/biodiversity/) end up being underwhelming
collections of reformatted journal articles, rather then innovative
integrations of biodiversity knowledge.  
  
With these issues in mind I\'ve started playing with *ZooKeys* XML,
initially looking at ways to display the article beyond the conventional
format. Ultimately I\'d like to embed the article in a broader web of
citations and data. *ZooKeys* articles are available in PDF, HTML, and
XML. The HTML has links to taxon pages, maps, etc., which is nice, but I
personally find this a little jarring because it interrupts the reading
experience. The *ZooKeys* web site also surrounds the article with all
paraphernalia of a publisher\'s web site:  
  
![Zookeys](http://lh5.ggpht.com/-JZnFYtcUgh4/Tu-fhL7FRgI/AAAAAAAABGw/cVGuFOK4wos/zookeys.png?imgmax=800 "zookeys.png"){style="display:block; margin-left:auto; margin-right:auto;"
border="0" width="450" height="298"}  
As a first experiment, I\'ve taken the XML for article **At the lower
size limit for tetrapods, two new species of the miniaturized frog genus
Paedophryne (Anura, Microhylidae)**
<http://dx.doi.org/10.3897/zookeys.154.1963> and used a XSLT style sheet
to reformat the article. I\'ve borrowed some ideas from Nature\'s apps,
such as the font for the title, displaying the abstract in bold, and
showing all the figures in the article as thumbnails near the top. I\'ve
also added some basic interactivity, which you can see in the video
below. Instead of figures being in one place in the article, wherever a
figure is mentioned in the article (e.g., \"Fig. 1\") if you click on
the reference to the figure it appears. If the article display a point
locality using latitude and longitude, instead of launching a separate
browser window with a Google map, click on the locality and the map
appears. The idea is that the flow of reading isn\'t interrupted,
figures, maps, and citations all appear in the text.  
  

::: {style="text-align:center;"}
::: {.iframe src="http://player.vimeo.com/video/33921915?title=0&byline=0&portrait=0&autoplay=0"}
:::
:::

  
This demo (which you can see live at <http://iphylo.org/~rpage/zookeys>)
is limited, but most of its functionality comes from simply reformatting
XML using XSLT. There\'s a little bit of jQuery for animation, and I
ended up having to write a PHP script to convert verbatim latitude and
longitude coordinates to the decimal coordinates expected by Google
Maps, but it\'s all very light weight. It wouldn\'t take much to add
some JSON queries to make the taxon names clickable (e.g., showing a
summary of a taxon from EOL). Because *ZooKeys* uses the NLM DTD for its
XML, some of this code could also be applied to other journals, such as
PLoS, so we could start to grow a library of linked, interactive
taxonomic articles.  


One of the things I keep revisiting is the way we display scientific articles. Apart from Nature's excellent iPhone and iPad apps, most efforts to re-imagine how we display articles are little more than glorified PDF viewers (e.g., the PLoS iPad app).Part of the challenge is that if we make the article more interactive we immediately confront the problem of how to link to other content.

Towards an interactive taxonomic article: displaying an article from ZooKeys

This post is simply a quick note on some experiments with DjVu that I haven't finished. Much of BHL's content is available as DjVu files, which contain both the scanned images and OCR text, complete with co-ordinates of each piece of text. This means that it would, in principle, be trivial to lay out the bounding boxes of each  text element on a web page.

DjVu XML to HTML

Tom Pasley recently drew my attention to CrossRef\'s addition of a [XML
format
parameter](http://www.crossref.org/CrossTech/2008/02/added_xml_format_parameter_to.html)
to their OpenURL resolver. Adding **&format=xml** to the OpenURL request
retrieves bibliographic metadata in \"unixref\" format (for those who
like this sort of thing, the XML schema is
[here](http://doi.crossref.org/doc/tech/unixref1.0.xsd)). The biggest
change is now the metadata lists more than one author for multi-author
papers.  
[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZaL2jrmibwIh4R4kDthiC6XPr9JZ4XbW4QYhIwSlpo9cBWr-hr-Vez1ScT1A_VqBG0wZ3tGTrpaY8TvVKYCqB1I_ZQSDZ43LtwzdXGJ4LjvWLI14MXiUBbBkvafl9SDDpsqyN/s320/TestXSLT.png){#BLOGGER_PHOTO_ID_5175749757460877410
style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;"
border="0"}](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZaL2jrmibwIh4R4kDthiC6XPr9JZ4XbW4QYhIwSlpo9cBWr-hr-Vez1ScT1A_VqBG0wZ3tGTrpaY8TvVKYCqB1I_ZQSDZ43LtwzdXGJ4LjvWLI14MXiUBbBkvafl9SDDpsqyN/s1600-h/TestXSLT.png){onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"}  
I tend to use JSON for my work now, so a common task is to convert XML
data streams into JSON. I\'ve modified my bioGUID OpenURL resolver to
make use of the unixref format, which meant I had to write a XSLT file
to convert unixref to JSON. If you\'re interested, you can grab a copy
[here](http://bioguid.info/xsl/unixref2JSON.xsl). It\'s not pretty, but
it seems to work OK.  
  
For some years now I\'ve relied on Marc Liyanage\'s excellent tool
[TestXSLT](http://www.entropy.ch/software/macosx/#testxslt) to develop
XSLT files. If you have a Mac and work with XSLT, then I do yourself a
favour and grab a copy of this free tool.


Tom Pasley recently drew my attention to CrossRef's addition of a XML format parameter to their OpenURL resolver.

Rogue Scholar Beiträge

Using xml schema and xslt in R

Towards an interactive taxonomic article: displaying an article from ZooKeys

DjVu XML to HTML

CrossRef adds more information to OpenURL resolver