Cameron Neylon

I think it is fair to say that even those of us most enamored of
post-publication peer review would agree that its effectiveness remains
to be demonstrated in a convincing fashion. Broadly speaking there are
two reasons for this; the first is the problem of social norms for
commenting. As in there aren't any. I think it was [Michael
Nielsen](http://michaelnielsen.org) who referred to the
"[Kabuki](http://en.wikipedia.org/wiki/Kabuki) Dance of scientific
discourse". It is entirely allowed to stab another member of the
research community in the back, or indeed the front, but there are
specific ways and forums in which it is acceptable to do. No-one quite
knows what the appropriate rules are for commenting on online fora, as
best [described most
recently](http://stevekochscience.blogspot.com/2009/06/my-first-rating-and-commenting-of-plos.html)
by [Steve Koch](http://stevekochscience.blogspot.com).

My feeling is that this is a problem that will gradually go away as we
evolve norms of behaviour in specific research communities. The current
"rules" took decades to build up. It should not be surprising if it
takes a few years or more to sort out an adapted set for online
interactions. The bigger problem is the one that is usually surfaced as
"I don't have any time for this kind of thing". This in turn can be
translated as, "I don't get any reward for this". Whether that reward is
a token for putting on your CV, actual cash, useful information coming
back to you, or just the warm feeling that someone else found your
comments useful, rewards are important for motivating people (and
researchers).

One of the things that links these two together is a sense of loss of
control over the comment. Commenting on journal web-sites is just that,
commenting on *the* *journal's website*. The comment author has "given
up" their piece of value, which is often not even citeable, but also
lost control over what happens to their piece of content. If you change
your mind, even if the site allows you to delete it, you have no way of
checking whether it is still in the system somewhere.

In a sense, when the Web 2.0 world was built it was got nearly precisely
wrong for personal content. For me [Jon Udell](http://blog.jonudell.net)
has written most clearly about this when he talks about the
publish-subscribe pattern for successful frameworks. In essence *I*
publish my content and you choose to subscribe to it. This works well
for me, the blogger, at this site, but it is not so great for the
commenter who has to leave their comment to my tender mercies on my
site. It would be better if the commenter could publish their comment
and I could *syndicate* it back to my blog. This creates all sorts of
problems; it is challenging for you to aggregate your own comments
together and you have to rely on the functionality of specific sites to
help you follow responses to your comments. Jon wrote about this better
than I can in [his blog
post](http://blog.jonudell.net/2009/03/06/hosted-lifebits-meets-infobus/).

So a big part of the problem could be solved if people streamed their
own content. This isn't going to happen quickly in the general sense of
everyone having a web server of their own -- it still remains too
difficult for even moderately skilled people to be bothered doing this.
Services will no doubt appear in the future but current broadcast
services like twitter offer a partial solution (its "my" twitter
account, I can at least pretend to myself that I can delete all of it).
The idea of using something like the twitter service at
[microrevie.ws](http://microrevie.ws/how-to) as
[suggested](http://friendfeed.com/science-2-0/5c1d0de0/sketch-of-rating-scheme-for-science-online-gave)
by [Daniel Mietchen](http://ways.org/en/blog/daniel) this week can go a
long way towards solving the problem. This takes a structured tweet of
the form @hreview {Object};{your review} followed optionally by a number
of asterisks for a star rating. This doesn't work brilliantly for papers
because of problems with the length of references for the paper, even
with shortened dois, the need for sometimes lengthy reviews and the
shortness of tweets. Additionally the twitter account is not
automatically associated with a unique research contributor ID. However
the principle of the author of the review controlling their own content,
while at the same time making links between themselves and that content
in a linked open data kind of way is extremely powerful.

Imagine a world in which your email outbox or local document store is
also webserver (via any one of an emerging set of tools like
[Wave](http://wave.google.com), [DropBox](http://getdropbox.com), or
[Opera Unite](http://unite.opera.com)). You can choose who to share your
review with and change that over the time. If you choose to make it
public the journal, or the authors can give you some form of credit. It
is interesting to think that author-side charges could perhaps be
reduced for valuable reviews. This wouldn't work in a naive way, with
\$10 per review, because people would churn outÂ  large amounts of
rubbish reviews, but if those reviews are out on the linked data web
then their *impact* can be measured by their page rank and the authors
rewarded accordingly.

Rewards and control linked together might provide a way of solving the
problem -- or at least of solving it faster than we are at the moment.


I think it is fair to say that even those of us most enamored of post-publication peer review would agree that its effectiveness remains to be demonstrated in a convincing fashion. Broadly speaking there are two reasons for this; the first is the problem of social norms for commenting. As in there aren’t any.

Pub-sub/syndication patterns and post publication peer review

University of Glasgow

Roderic Page

*PLoS Computational Biolgy* has recently published \"Adventures in
Semantic Publishing: Exemplar Semantic Enhancements of a Research
Article\"
([doi:10.1371/journal.pcbi.1000361](http://dx.doi.org/10.1371/journal.pcbi.1000361))
by David Shotton and colleagues. As a proof of concept, they took Reis
et al.
([doi:10.1371/journal.pntd.0000228](http://dx.doi.org/10.1371/journal.pntd.0000228))
and \"semantically enhanced\" it:  

> These semantic enhancements include provision of live DOIs and
> hyperlinks; semantic markup of textual terms, with links to relevant
> third-party information resources; interactive figures; a re-orderable
> reference list; a document summary containing a study summary, a tag
> cloud, and a citation analysis; and two novel types of semantic
> enrichment: the first, a Supporting Claims Tooltip to permit
> "Citations in Context", and the second, Tag Trees that bring together
> semantically related terms. In addition, we have published
> downloadable spreadsheets containing data from within tables and
> figures, have enriched these with provenance information, and have
> demonstrated various types of data fusion (mashups) with results from
> other research articles and with Google Maps.

The enhanced article is here:
[doi:10.1371/journal.pntd.0000228.x001](http://dx.doi.org/10.1371/journal.pntd.0000228.x001).
For background on these enhancements, see also David\'s companion
article \"Semantic publishing: the coming revolution in scientific
journal publishing\"
([doi:10.1087/2009202](http://dx.doi.org/10.1087/2009202), PDF preprint
[available
here](http://imageweb.zoo.ox.ac.uk/pub/2008/publications/Shotton_Semantic_publishing_evaluation.pdf)).
The process is summarised in the figure below (Fig. 10 from Shotton et
al.,
[doi:10.1371/journal.pcbi.1000361.g010](http://dx.doi.org/10.1371/journal.pcbi.1000361.g010)).  
  
[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKyPcHllndJu-4DHOIPJA3djD1OCUhTiN9-YCzt8adBKivM-zEtL7RuOuL9AKenIO994rQ6SATVMWdhcml4tqxxw1BW-Vu4OtFX2FpTxyfa_SE5M1vu1rid0y1h5bzSXX4qau5/s400/journal.pcbi.1000361.g010.png){#BLOGGER_PHOTO_ID_5326644951892583362
style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 306px;"
border="0"}](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKyPcHllndJu-4DHOIPJA3djD1OCUhTiN9-YCzt8adBKivM-zEtL7RuOuL9AKenIO994rQ6SATVMWdhcml4tqxxw1BW-Vu4OtFX2FpTxyfa_SE5M1vu1rid0y1h5bzSXX4qau5/s1600-h/journal.pcbi.1000361.g010.png){onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"}  
  
While there is lots of cool stuff here (see also Elsevier\'s [Article
2.0 Contest](http://article20.elsevier.com/), and the [Grand
Chalenge](http://www.elseviergrandchallenge.com/), for which David is
one of the judges), I have a couple of reservations.  
  
**The unique role of the journal article?**  
  
Shotton et al. argue for a clear distinction between journal article and
database, in contrast to the view articulated by Philip Bourne
([doi:10.1371/journal.pcbi.0​010034](http://dx.doi.org/10.1371/journal.pcbi.0%E2%80%8B010034))
that there\'s really no difference between a database and a journal
article and that the two are converging. I tend to favour the later
viewpoint. Indeed, as I argued in my Elsevier Challenge entry
([doi:10.1038/npre.2008.2579.1](http://dx.doi.org/10.1038/npre.2008.2579.1)),
I think we should publish articles (and indeed data) as wikis, so that
we can [fix the inevitable
error](http://iphylo.blogspot.com/2008/11/sequencing-carmen-electra.html).
We can always roll back to the original version if we want to see the
author\'s original paper.  
  
**Real linking**  
  
But my real concern is that the example presented is essentially
\"integration by linking\", that is, the semantically enhanced version
gives us lots of links to other information, but these are regular
hyperlinks to web pages. So, essentially we\'ve gone from pre-web
documents with no links, to documents where the bibliography is
hyperlinked (most online journals), to documents where both the
bibliography and some terms in the text are hyperlinked (a few journals,
plus the Shotton [et al.]{.Apple-style-span style="font-style: italic;"}
example). I\'m a tad underwhelmed.  
[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfYfM7aHo4ivxTTk3prSSsMrWW0cn0xD8tGWaINqfVzlo3c6DOLMTBl2s7Xj7cacwch0xYOsmH6_xoCleKAc5OQ-lGhZJNhlA8THOsmWgp3k_ax8jov_4Fs6CwNeVKQNq-Cj0t/s400/shotton1.png){#BLOGGER_PHOTO_ID_5326656888294050114
style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 248px; height: 400px;"
border="0"}](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfYfM7aHo4ivxTTk3prSSsMrWW0cn0xD8tGWaINqfVzlo3c6DOLMTBl2s7Xj7cacwch0xYOsmH6_xoCleKAc5OQ-lGhZJNhlA8THOsmWgp3k_ax8jov_4Fs6CwNeVKQNq-Cj0t/s1600-h/shotton1.png){onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"}What
bothers me about this is:

1.  The links are to web pages, so it will be hard to do computation on
    these (unless the web page has easily retrievable metadata)
2.  There is no reciprocal linking \-- the resource being linked to
    doesn\'t know it is the target of the link

  
  
***Web pages are for humans***  
  
The first concern is that the marked-up article is largely intended for
human readers. Yes, there are associated metadata files in RDF N3, but
the core \"added value\" is really only of use to humans. For it to be
of use to a computer, the links would have to go to resource that the
computer can understand. A human clicking on many of the links will get
a web page and they can interpret that, but computers are thick and they
need a little help. For example, one hyperlinked term is [*Leptospira*
spirochete](http://www.ubio.org/browser/details.php?namebankID=2556590),
linked to the uBio namebank record (click on the link to see it). The
link resolves to a web page, so it\'s not much use to a computer (unless
if has a scrapper for uBio HTML). Ironically, uBio serves LSIDs, so we
could retrieve RDF metadata for this name
([urn:lsid:ubio.org:namebank:255659](http://bioguid.info/urn:lsid:ubio.org:namebank:2556590)),
but there\'s nothing in the uBio web page that tells the computer
that.  
  
Of course, Shotton *et al.* aren\'t responsible for the fact that most
web pages aren\'t easily interpreted by computers, but simply embedding
links to web pages isn\'t a big leap forward. What could they have done
instead? One approach is to link to resources that are
computer-readable. For example, instead of linking the term \"Oswaldo
Cruz Foundation\" to that organisation\'s home page
(<http://www.fiocruz.br/cgi/cgilua.exe/sys/start.htm?tpl=home>), why not
use the DBpedia URI <http://dbpedia.org/page/Instituto_Oswaldo_Cruz>?
Now we get both a human-readable page, and extensive RDF that a computer
can use. In other words, if we crawl the semantically enhanced *PLoS*
article with a program, I want to be able to have that crawler follow
the links and still get useful information, not the dead end of a HTML
web page. Quite a few of the institutions listed in the enhanced paper
have DBPedia URIs:  
  

- Brazilian National Commission for Ethics in Research
- [Brazilian National Research
  Council](http://dbpedia.org/page/Brazilian_National_Institute_for_Space_Research)
- [Centers for Diseases Control and
  Prevention](http://dbpedia.org/page/Centers_for_Disease_Control_and_Prevention)
- Centro de Pesquisas Gonçalo Moniz, Fundação Oswaldo Cruz, Ministério
  da Saúde
- Company for Urban Development of the State of Bahia (CONDER)
- Division of International Medicine and Infectious Diseases, Weill
  Medical College of Cornell University
- [Environmental Systems Research
  Institute](http://dbpedia.org/page/ESRI)
- Escola Nacional da Saúde Pública, Fundação Oswaldo Cruz, Ministério da
  Saúde
- Image Bioinformatics Research Group, Department of Zoology, University
  of Oxford
- MRC Biostatistics Unit
- [Oswaldo Cruz
  Foundation](http://dbpedia.org/page/Instituto_Oswaldo_Cruz)
- R Foundation for Statistical Computing
- Royal Tropical Institute, Holland
- Secretária Estadual de Saúde da Bahia
- [Universidad de Buenos
  Aires](http://dbpedia.org/page/Universidad_de_Buenos_Aires)
- Universidade Federal Rural do Rio de Janeiro
- Urban Health Council of Pau da Lima
- [US National Institutes of
  Health](http://dbpedia.org/page/National_Institutes_of_Health)
- [Weill Medical College of Cornell
  University](http://dbpedia.org/page/Weill_Cornell_Medical_College_of_Cornell_University)
- WHO Collaborative Laboratory for Leptospirosis

  
Why does this matter? Well, if you use DBPedia URIs you get RDF, plus
you get connections with the [Linked Data](http://www.linkeddata.org/)
crowd, who are rapidly linking diverse data sets together:  
[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3TgLEfnov-sT62v30YeoprIGiW6lOs70p30pgvT7ccXppUCc1zMzAgH9nMMf_FsdypbyzUdcVCE68UOJILvjWnFtT9uiGHBAYFfRCppXh_OrlXJCBLNIzxVRDdf5pSJYekebH/s400/lod-datasets_2009-03-05-scaled.png){#BLOGGER_PHOTO_ID_5323331608339372786
style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 305px;"
border="0"}](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3TgLEfnov-sT62v30YeoprIGiW6lOs70p30pgvT7ccXppUCc1zMzAgH9nMMf_FsdypbyzUdcVCE68UOJILvjWnFtT9uiGHBAYFfRCppXh_OrlXJCBLNIzxVRDdf5pSJYekebH/s1600-h/lod-datasets_2009-03-05-scaled.png){onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"}  
  
I think this is where we need to be headed, and with a little extra
effort we can get there, once we move on from thinking solely about
human readers.  
  
An alternative approach (and one that I played with in my [Challenge
entry](http://iphylo.org/~rpage/challenge), as well as my ongoing wiki
efforts) is to create what Vandervalk et al. term a \"semantic
warehouse\"
([doi:10.1093/bib/bbn051](http://dx.doi.org/10.1093/bib/bbn051)).
Information about each object of interest is stored locally, so that
clicking on a link doesn\'t take you off-site into the world wide
wilderness, but to information about that object. For example, the page
for the paper [Mitochondrial paraphyly in a polymorphic poison frog
species (Dendrobatidae; D.
pumilio)](http://iphylo.org/~rpage/challenge/www/uri/50c76f37adcb535ddfa9ae6544ee3f2f)
lists the papers cited, clicking on one takes you to the page about that
paper. There are limitations to this approach as well, but the key thing
is that one could imagine doing computations over this (e.g., computing
citation counts for DNA sequences, or geospatial queries across papers)
that simple HTML hyperlinking won\'t get you.  
  
***Reciprocal links***  
  
The other big issue I have with the Shotton *et al.* \"integration by
linking\" is that it is one-way. The semantically enhanced paper
\"knows\" that it links to, say, the uBio record for
[*Leptospira*](http://www.ubio.org/browser/details.php?namebankID=2556590),
but uBio doesn\'t know this. It would enhance the uBio record if it knew
that
[doi:10.1371/journal.pntd.0​000228.x001](http://dx.doi.org/10.1371/journal.pntd.0​000228.x001)
linked to it.  
  
Links are inherently reciprocal, in the sense that if paper 1 cites
paper 2, then paper 2 is cited by paper 1.  
[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1Es03-lg_yA5gKgqzatNKNHWwmuPYcvBJ0AxsaMg4bNwkLNQuZg536X9nYklvCt6sYyKr84cc7GedD4g5okbdjKeDHy3Fae_bZ4JT3zsQMs4oug6Sd1qsi-3YUwgdv5tLLbva/s400/reciprocal.png){#BLOGGER_PHOTO_ID_5326677732139091538
style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 383px; height: 295px;"
border="0"}](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1Es03-lg_yA5gKgqzatNKNHWwmuPYcvBJ0AxsaMg4bNwkLNQuZg536X9nYklvCt6sYyKr84cc7GedD4g5okbdjKeDHy3Fae_bZ4JT3zsQMs4oug6Sd1qsi-3YUwgdv5tLLbva/s1600-h/reciprocal.png){onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"}  
Publishers understand this, and the web page of an article will often
show lists of papers that cite the paper being displayed. How do we do
this for data and other objects of interest? If we database everything,
then it\'s straightforward. CrossRef is storing citation metadata and
offers a \"forward linking\" service, some publishers (e.g., Elsevier
and Highwire) offer their own versions of this. In the same way, this
record for GenBank sequence
[AY322281](http://iphylo.org/~rpage/challenge/www/uri/3593f20071b4839185eb4a237146690e)
\"knows\" that it is cited by (at least) two papers because I\'ve stored
those links in a database. Knowing that you\'re being linked to
dramatically enhances discoverability. If I\'m browsing uBio I gain more
from the experience if I know that the *PLoS* paper cites
[*Leptospira*](http://www.ubio.org/browser/details.php?namebankID=2556590).  
  
***Knowing when you\'re being linked to***  
  
If we database everything locally then reciprocal linking is easy. But,
realistically, we can\'t database everything (OK, maybe that\'s not
strictly true, can can think of Google as a database of everything). The
enhanced *PLoS* paper \"knows\" that it cites the uBio record, how can
the uBio record \"know\" that it has been cited by the *PLoS* paper?
What if the act of linking was reciprocal? How can we achieve this in a
distributed world? Some possibilities:  

- we have an explicit API embedded in the link so that uBio can extract
  the source of the link (could be spoofed, need authentication?)
- we use OpenURL-style links that embed the *PLoS* DOI, so that uBio
  knows the source of the link (OpenURL is a mess, but potentially very
  powerful)
- uBio uses the HTTP referrer header to get the source of the link, then
  parses the *PLoS* HTML to extract metadata and the DOI (ugly screen
  scraping, but no work for *PLoS*)

  
Obviously this needs a little more thought, but I think that **real**
integration by linking requires that the resources being linked are both
computer and human readable, and that **both** resources know about the
link. This would create much more powerful \"semantically enhanced\"
publications.


<em>
 PLoS Computational Biolgy
</em>
has recently published "Adventures in Semantic Publishing: Exemplar Semantic Enhancements of a Research Article" (doi:10.1371/journal.pcbi.1000361) by David Shotton and colleagues. As a proof of concept, they took Reis et al. (doi:10.1371/journal.pntd.0000228) and "semantically enhanced" it:  The enhanced article is here: doi:10.1371/journal.pntd.0000228.x001.

Semantic Publishing: towards real integration by linking

This post is an opinion piece and not a rigorous objective analysis. It
is fair to say that I am on the record as and [advocate of the
principles behind PLoS
ONE](http://blog.openwetware.org/scienceintheopen/2008/07/13/what-i-missed-on-my-holiday-or-why-i-like-refereeing-for-plos-one/)
and am also in favour of post publication peer review and this should be
read in that light. \[*ed I've also modified this slightly from the
original version because I got myself mixed up in an Excel
spreadsheet\]*

To me, anonymous peer review is, and always has been, broken. The
central principle of the scientific method is that claims and data to
support those claims are placed, publically, in the view of expert
peers. They are examined, and re-examined on the basis of new data,
considered and modified as necessary, and ultimately discarded in favour
of an improved, or more sophisticated model. The strength of this
process is that it is open, allowing for extended discussion on the
validity of claims, theories, models, and data. It is a bearpit, but one
in which actions are expected to take place in public (or at least
community) view. To have as the first hurdle to placing new science in
the view of the community a process which is confidential, anonymous,
arbitrary, and closed, is an anachronism.

It is, to be fair, an anachronism that was necessary to cope with rising
volumes of scientific material in the years after the second world war
as the community increased radically in size. A limited number of
referees was required to make the system manageable and anonymity was
seen as necessary to protect the integrity of this limited number of
referees. This was a good solution given the technology of the day.
Today, it is neither a good system, nor an efficient system, and we have
in principle the ability to do peer review differently, more
effectively, and more efficiently. However, thus far most of the
evidence suggests that the scientific community dosen't want to change.
There is, reasonably enough, a general attitude that if it isn't broken
it doesn't need fixing. Nonetheless there is a constant stream of
suggestions, complaints, and experimental projects looking at
alternatives.

The last 12-24 months have seen some radical experiments in peer review.
Nature Publishing Group [trialled an open peer review
process](http://www.nature.com/nature/peerreview/). PLoS ONE proposed a
qualitatively different form of peer reivew, rejecting the idea of
'importance' as a criterion for publication.
[Frontiers](http://www.frontiersin.org/neuroscience/) have developed [a
tiered
approach](http://admin.frontiersin.org/neuroscience/FrontiersPublishingProcess.do)
where a paper is submitted into the 'system' and will gradually rise to
its level of importance based on multiple rounds of community review.
[Nature Precedings](http://precedings.nature.com) has expanded the role
and discipline boundaries of pre-print archives and a [white
paper](https://wiki.systemsx.ch/display/SPW2E/Draft+White+Paper) has
been presented to [EMBO](http://www.embo.org/) Council suggesting that
the majority of EMBO journals be scrapped in favour of retaining one
flagship journal for which papers would be handpicked from a generic
repository where authors would submit, along with referees' reports and
author's response, on payment of a submission charge. Of all of these
experiments, none could be said to be a runaway success so far with the
possible exception of PLoS ONE. PLoS ONE, as I have [written
before](http://blog.openwetware.org/scienceintheopen/2008/07/13/what-i-missed-on-my-holiday-or-why-i-like-refereeing-for-plos-one/),
succeeded precisely because it managed to reposition the definition of
'peer review'. The community have accepted this definition, primarily
because it is indexed in [PubMed](http://pubmed.org). It will be
interesting to see how this develops.

PLoS has also been aiming to develop ratings and comment systems for
their papers as a way of moving towards some element of post publication
peer review. I, along with some others (see full disclosure below) have
been granted access to the full set of comments and some analytical data
on these comments and ratings. This should be seen in the context of
[Euan Adie's discussion of commenting frequency and
practice](http://blogs.nature.com/wp/nascent/2008/07/who_leaves_comments_on_scienti_1.html)
in BioMedCentral journals which broadly speaking showed that around 2%
of papers had comments and that these comments were mostly substantive
and dealt with the science. How does PLoS ONE compare and what does this
tell us about the merits or demerits of post publication peer review?

PLoS ONE has a range of commenting features, including a simple rating
system (on a scale of 1-5) the ability to leave freetext notes,
comments, and questions, and in keeping with a general Web 2.o feel the
ability to add trackbacks, a mechanism for linking up citations from
blogs. Broadly speaking a little more than 13% (380 of 2773) of all
papers have ratings and around 23% have comments, notes, or replies to
either (647 of 2773, not including any from PLoS ONE staff) . Probably
unsurprisingly most papers that have ratings also have comments. There
is a very weak positive correlation between the number of citations a
paper has received (as determined from Google Scholar) and the number of
comments (R\^2 = 0.02, which is probably dominated by papers with both
no citations and no comments, which are mostly recent, none of this is
controlled for publication date).

Overall this is consistent with what we'd expect. The majority of papers
don't have either comments or ratings but a significant minority do.
What is slightly suprising is that where there is arguably a higher
barrier to adding something (click a button to rate versus write a text
comment) there is actually more activity. This suggests to me that
people are actively uncomfortable with rating papers versus leaving
substantive comments. These numbers compare very favourably to those
reported by Euan on comments in BioMedCentral but they are not yet
moving into the realms of the majority. It should also be noted that
there has been a consistentÂ  programme at PLoS ONE with the aim of
increasing the involvement of the community. Broadly speaking I would
say that the data we have suggest that that programme has been a success
in raising involvement.

So are these numbers 'good'? In reality I don't know. They seem to be an
improvement on the BMC numbers arguing that as systems improve and
evolve there is more involvement. However, one graph I received seems to
indicate that there isn't an increase in the frequency of comments
within PLoS ONE over the past year or so which one would hope to see.
Has this been a radical revision of how peer review works? Well not yet
certainly, not until the vast majority of papers have ratings, but more
importantly not until we have evidence that people are *using* those
ratings. We are not yet in a position where we are about to see a
stampede towards radically changed methods of peer review and this is
not surprising. Tradition changes slowly -- we are still only just
becoming used to the idea of the 'paper' being something that goes
beyond a pdf, embedding that within a wider culture of online rating and
the use of those ratings will take some years yet.

So I have spent a number of posts recently discussing the details of how
to make web services better for scientists. Have I got anything useful
to offer to PLoS ONE? Well I think [some of the criteria I suggested
last
week](http://cameronneylon.net/blog/how-to-make-connotea-a-killer-app-for-scientists/)
might be usefully considered. The problem with rating is that it lies
outside the existing workflow for most people. I would guess that many
users don't even see the rating panel on the way into the paper. Why
would people log into the system to look at a paper? What about making
the rating implicit when people bookmark a paper in external services?
Why not actually use that as the rating mechanism?

I emphasised the need for a service to be useful to the user before
there are any 'social effects' present. What can be offered to make the
process of rating a paper useful to the single user in isolation? I
can't really see why anyone would find this useful unless they are
dealing with huge number of papers and can't remember which one is which
from day to day. It may be useful within groups or journal clubs but all
of these require a group to sign up.Â  It seems to me that if we can't
frame it as a useful activity for a single person then it will be
difficult to get the numbers required to make this work effectively on a
community scale.

In that context, I think getting the numbers to around the 10-20% level
for either comments or ratings has to be seen as an immense success. I
think it shows how difficult it is to get scientists to change their
workflows and adopt new services. I also think there will be a lot to
learn about how to improve these tools and get more community
involvement. I believe strongly that we need to develop better
mechanisms for handling peer review and that it will be a very difficult
process getting there. But the results will be seen in more efficient
dissemination of information and more effective communication of the
details of the scientific process. For this PLoS, the PLoS ONE team, as
well as other publishers, including BioMedCentral, Nature Publishing
Group, and others, that are working on developing new means of
communication and improving the ones we have deserve applause. They may
not hit on the right answer first off, but the current process of
exploring the options is an important one, and not without its risks for
any organisation.

*Full disclosure: I was approached along with a number of other bloggers
to look at the data provided by PLoS ONE and to coordinate the release
of blog posts discussing that data. At the time of writing I am not
aware of who the other bloggers are, nor have I read what they have
written. The data that was provided included a list of all PLoS ONE
papers up until 30 July 2008, the number of citations, citeulike
bookmarks, trackbacks, comments, and ratings for each paper. I also
received a table of all comments and a timeline with number of comments
per month. I have been asked not to release the raw data and will honour
that request as it is not my data to release. If you would like to see
the underlying data please get in contact with Bora Zivkovic*.


This post is an opinion piece and not a rigorous objective analysis. It is fair to say that I am on the record as and advocate of the principles behind PLoS ONE and am also in favour of post publication peer review and this should be read in that light. [
<em>
 ed I’ve also modified this slightly from the original version because I got myself mixed up in an Excel spreadsheet]
</em>
To me, anonymous peer review is, and always has been, broken.

Postagens de Rogue Scholar

Pub-sub/syndication patterns and post publication peer review

Semantic Publishing: towards real integration by linking

Can post publication peer review work? The PLoS ONE report card