Lars Willighagen

I made a test JavaScript file for NodeJS to practice working with it,
and prepare more for the project.  
  
The result
(code [here](https://github.com/larsgw/contentmine-fellowship/blob/master/js/test.js))
is a small program with some options:  
  
  

    Options:

      -h, --help     output usage information
      -V, --version  output the version number
      -o       File to write data to
      -c       File to print
      -t     Text to print
      -d     Data to write to file
      -v     Verbosity: DEBUG, INFO, LOG, WARN or ERROR

I made a custom logger and a program to parse passed arguments. The
option to parse with the commander module, as seen here:  

    var program = require(commander);

    program.parse(process.argv);

which is used in the `quickscrape` code as well, didn\'t really work for
me. Maybe I did something wrong. When I run the following code  

    var program = require('commander');

    program
      .option('-t',
          'Test A')
      .option('-T',
          'Test B')
     .parse(process.argv);

    console.log(program);

like this  

    node file.js -t 'Test_A' -T 'Test_B'

the console outputs this:  

    { commands: [],
      options: 
      [ { flags: '-t',
    required: 0,
    optional: 0,
    bool: true,
    long: '-t',
    description: 'Test A' },
        { flags: '-T',
      required: 0,
      optional: 0,
      bool: true,
      long: '-T',
      description: 'Test B' } ],
      _execs: {},
      _allowUnknownOption: false,
      _args: [],
      _name: 'file',
      Command: [Function: Command],
      Option: [Function: Option],
      _events: { '-t': [Function], '-T': [Function] },
      rawArgs: 
      [ 'node',
        '/home/larsw/public_html/cm-repo/js/file.js',
        '-t',
        'Test_A',
        '-T',
        'Test_B' ],
      T: true,
      args: [ 'Test_A', 'Test_B' ] }

This doesn\'t seem right to me. If is understand correctly, `'Test_A'`
is passed to `-t` and `'Test_B'` to `-T`. Instead, `commander` seems to
say `-t` and/or `-T` passed (`T: true` is present when you pass only
`-t` too), and two strings are passed without context. Maybe there\'s
something wrong with the `program.option()` things. Maybe my knowledge
about command line arguments is incorrect, but I don\'t think to this
extent. Just the same, I made a parser myself. Not sure if it follows
any standard, but it works for me, for now. I will update the
documentation shortly.  
  
Next week I\'ll look into `norma` to see how it changes the format of
papers from XML/HTML to sHTML, and ChemicalTagger to see how it
recognises sentences.


I made a test JavaScript file for NodeJS to practice working with it, and prepare more for the project.

Weekly Report 3: Playing with NodeJS

I updated [Citation.js](https://larsgw.github.io/citation.js/) to accept
`quickscrape`\'s results.json. Results can be viewed
[here](https://larsgw.github.io/citation.js/demo.html#cm-in).  
  
I made a card prototype for the database HTML page, using data extracted
with quickscrape. It\'s mainly, although not advanced, a design thing,
but it\'s nice to see what information I can fit. Results are
[here](https://larsgw.github.io/contentmine-fellowship/html/card.html).  
  

|                                                                                                                                                                                                                                                                                                                                                                                                         |
|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
| [![](https://3.bp.blogspot.com/-Gd2lLm8tU8c/V6n8TarX44I/AAAAAAAALv8/POHEYrKFSI05P6X5oGtqhOfodxWTYbLNQCPcB/s320/paper-column1.png){border="0" height="320" width="141"}](https://3.bp.blogspot.com/-Gd2lLm8tU8c/V6n8TarX44I/AAAAAAAALv8/POHEYrKFSI05P6X5oGtqhOfodxWTYbLNQCPcB/s1600/paper-column1.png){imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"} |
|                                                                                                                                                                                            With opened paper                                                                                                                                                                                            |

|                                                                                                                                                                                                                                                                                                                                                                                                                          |
|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
| [![](https://1.bp.blogspot.com/-ICdSXWzibFM/V6n8TQ83CpI/AAAAAAAALwA/zozRnuUkyeYvlcJ45iyPKRpnPIxXOBhjACPcB/s320/paper-column.png){border="0" height="320" width="188"}](https://1.bp.blogspot.com/-ICdSXWzibFM/V6n8TQ83CpI/AAAAAAAALwA/zozRnuUkyeYvlcJ45iyPKRpnPIxXOBhjACPcB/s1600/paper-column.png){imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto; text-align: center;"} |
|                                                                                                                                                                                                   Without opened paper                                                                                                                                                                                                   |

  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
I have taken a look at quickscrape. What I\'d like to do is to practise
building a similar program myself, first just outputting strings to the
command line, later processing arguments, to eventually use in my
project.  
  
I\'ll probably try to do that next week.


I updated Citation.js to accept quickscrape's results.json. Results can be viewed here. I made a card prototype for the database HTML page, using data extracted with quickscrape. It's mainly, although not advanced, a design thing, but it's nice to see what information I can fit. Results are here. With opened paper Without opened paper  I have taken a look at quickscrape.

Weekly Report 2: Final project prototype

## Extended proposal ( 3 June 2016, [GitHub](https://github.com/larsgw/contentmine-fellowship/wiki/Research-project), with comments ): Analyzer and data visualizer for papers about conifers {#extended-proposal-3-june-2016-github-with-comments-analyzer-and-data-visualizer-for-papers-about-conifers}

> ### Description
>
> A program visualizing and listing data extracted with ContentMine
> software from a list of papers resulting from a query. Different types
> of data (authors, genes, molecules etc.) will be listed in different
> columns.  
>   
>
> ### My proposal
>
> My proposal is to collect data about conifers and visualize it in a
> dynamic HTML page. It consists of three parts.  
>   
> The first part is to make and/or generate the RegEx after analyzing
> which sentences occur often. Examples of such sentences are ` is `,
> ` has ` and `After , displays `. Last example from
> [PMC4351838](http://dx.doi.org/10.1186/s12870-015-0447-z) (Original
> sentence: *\'Conversely, after girdling, Pinus canariensis displays an
> active growth from the upper edge, being often able to reconnect the
> phloem and surmount the injury if the removed ring is not too
> wide.\'*).  
>   
> Selecting sentence structures from the text will be done manually, as
> it would take way more time and work to write a program for this
> purpose. Then, actual expressions can be generated to match for
> certain species, by combining RegEx to match a certain name and RegEx
> for the found sentence structures.

  
See also the comment about the first part of the process in the more
detailed description.  
  

> The second part is to write a program to convert all data to JSON
> format. The structure of the JSON will, if possible, follow certain
> guidelines. An example of a guideline is the structure Wikidata uses.
> If not possible, guidelines will be made. The data will periodically
> be converted by hand, and put on a server.

  
This will probably be done by a custom NodeJS program.  
  

> The third part is to use the data to make a database. This will be
> shown on a dynamic website, which will probably be made with HTML, CSS
> and jQuery or AngularJS and will be put online. Its main structure is
> explained here:  
>   
>
> - It will have a search function. The results will be a list of
>   objects found in the database, and all related items.
> - The object will be an expandable card containing information like
>   type of object (species, gene, etc.), graphs and properties found
>   with the RegEx.
> - The information will link cards of the corresponding objects.
>
>   

  
The following part is most of the extension.  
  

> A more detailed description of the process:  
>   
>
> ### Input
>
> Papers will be fetched manually with a `getpapers` query. This query
> will look for the occurrence of several conifer species in the
> abstract, to further ensure the paper is about the species, and not
> just mentioning it. The full text, if available, will be downloaded as
> XML, because this is easier to analyse. Example of the query:  
>   
>
>     getpapers -q 'ABSTRACT:"Pinus sylvestris"' -o Pinus_sylvestris -x
>
>   
> This downloads the XML files (`-x`) and metadata of papers with the
> word combination \"Pinus sylvestris\" in the abstract
> (`-q 'ABSTRACT:"Pinus sylvestris"'`) into the output directory named
> Pinus_sylvestris (`-o Pinus_sylvestris`)  
>   
> When the papers are downloaded they will be, again manually, formatted
> to sHMTL (scholarly html) with the `norma` command. Example of the
> command:  
>   
>
>     norma --project Pinus_sylvestris -i fulltext.xml -o scholarly.html --transform nlm2html
>
>   
> Then, the formatted HTML is analysed with the `ami` package, and the
> data is stored. For example, the binomial species are extracted like
> this:  
>   
>
>     ami2-species --project Pinus_sylvestris -i scholarly.html --sp.species --sp.type binomial

  
Not that interesting or special, so this bit was shortened in the normal
proposal.  
  

> ### Further processing
>
> The next step is to automatically find out the relation between the
> found data and the species. For this to work, you need RegEx, or an
> entire program, that can recognize often occurring sentences that says
> something about the species. For example, a sentence like this (from
> [PMC4422480](http://dx.doi.org/10.1186%2Fs12864-015-1546-9/)):  
>   
>
>     ...it produces a number of important metabolites, e.g. flavonoids, anthocyanins, stilbenes, condensed tannins and phenolics.
>
>   
> can be analysed with RegEx. This uses sentences with certain
> structures to find properties of, in this case, Scots pine. Both the
> property name, and possibly the value, are stored, and used in further
> processing. Other examples of such sentences are ` has `, ` is ` and
> `After , displays `. Last example from
> [PMC4351838](http://dx.doi.org/10.1186%2Fs12870-015-0447-z/), original
> sentence: \'... after girdling, Pinus canariensis displays an active
> growth from the upper edge \...\'  
>   
> Selecting sentence structures from the text will be done manually, as
> it would take way more time and work to write a program for this
> purpose. Then, actual expressions can be generated to match for
> certain species, by combining RegEx to match a certain name (for
> example `P(inus|\.) sylvestris`) and RegEx for the found sentence
> structures.

  
As I said here, RegEx is not the only option, but at the time it seemed
the most viable one. However, since I read the papers about OSCAR,
ChemicalTagger, and SyntaxNet, such programs seem to be a better option,
as described in [Weekly Report
4](https://larsgw.blogspot.nl/2016/08/weekly-report-4.html).  
  

> ### Output
>
> The properties of the species and other objects found in the papers
> will be stored on a webserver as JSON. This way, a database for
> conifers can be generated which can be searched by users. Search
> results may include tables, graphs and Wikidata-like pages.  
>   
>
> ### HTML page
>
> The exact structure of the page is still to be discussed, but will
> probably be made with HTML, CSS and jQuery or AngularJS. It will fetch
> the JSON data from the server, and when the user searches, generate
> graphs and display data.  
>   
>
> |                                                                                                                                                                                                              |
> |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
> | ![Example of a column, here with tweets.](https://github.com/larsgw/contentmine-fellowship/raw/master/img/Example_tweet_column.png){height="320" style="margin-left: auto; margin-right: auto;" width="180"} |
> |                                                                                    Example of a column, here with tweets.                                                                                    |

  
A more recent design concept can be viewed in [Weekly Report
2](https://larsgw.blogspot.nl/2016/07/weekly-report-2.html).  
  

> ### Outreach
>
> I will blog regularly what I\'ve accomplished and how. This will be
> done with reports, probably occurring every one or two weeks,
> containing examples and references to code. I\'ll use this GitHub
> repository to keep track of my own tasks and resulting code, and the
> wiki for reports.

  
And on the blog now as well.  
  

## Original proposal

  
[This](https://drive.google.com/file/d/0B51JKcEdNx3pSVg5NEE0M2xuYmc/view?usp=sharing)
is the proposal I applied with.


Extended proposal ( 3 June 2016, GitHub, with comments ): Analyzer and data visualizer for papers about conifers   See also the comment about the first part of the process in the more detailed description. This will probably be done by a custom NodeJS program. The following part is most of the extension. Not that interesting or special, so this bit was shortened in the normal proposal.

Research project proposals

Start of the project. What I am going to do:  

- update [Citation.js](https://larsgw.github.io/citation.js/) to accept
  `quickscrape`\'s results.json
- look into `quickscrape` and the other programs, mainly to see how I
  can make a similar command for my own project
- perhaps add/improve some scrapers
- try to make a card prototype for the database

No real work yet, but some orientation and preparing until I have some
more information.  
  

|                                                                                                                                                                                                                                                                                                                            |
|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
| [![Citation.js](https://4.bp.blogspot.com/-2xAXhdo7L_M/V6n2O2LUtDI/AAAAAAAALvs/36OW2gIS2G0y48dukqriUr-DqNfOVivRACLcB/s320/slider_img-02.png){border="0" height="170" width="320"}](https://larsgw.github.io/citation.js/){imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"} |
|                                                                                                                                                        Citation.js                                                                                                                                                         |

  


Start of the project. What I am going to do: update Citation.js to accept quickscrape's results.json look into quickscrape and the other programs, mainly to see how I can make a similar command for my own project perhaps add/improve some scrapers try to make a card prototype for the database  No real work yet, but some orientation and preparing until I have some more information.

Messaggi di Rogue Scholar

Weekly Report 3: Playing with NodeJS

Weekly Report 2: Final project prototype

Research project proposals

Weekly Report 1: Start of my ContentMine fellowship