Cameron Neylon

I have long being sceptical of the costs and value delivered by our
traditional methods of peer review. This is really on two fronts,
firstly that the costs, [where they have been
estimated](http://www.rin.ac.uk/rin-model/) are extremely high,
representing a multi-billion dollar subsidy by governments of the
scholarly publishing industry. Secondly the value that is delivered
through peer review, the critical analysis of claims, informed opinion
on the quality of the experiments, is largely lost. At best it is
wrapped up in the final version of the paper. At worst it is simply
completely lost to the final end user. A part of this, which the more I
think about the more I find bizarre is that the whole process is carried
on under a shroud of secrecy. This means that as an end user, as I do
not know who the peer reviewers are, and do not necessarily know whatÂ 
process has been followed or even the basis of the editorial decision to
publish. As a result I have no means of assessing the quality of peer
review for any given journal, let alone any specific paper.

Those of us who see this as a problem have a responsibility to provide
credible and workable alternatives to traditional peer review. So far
despite many ideas we haven't, to be honest, had very much success.
Post-publication commenting, [open peer
review](http://www.nature.com/nature/peerreview/), and
[Digg](http://digg.com) like voting mechanisms have been explored but
have yet to have any large success in scholarly publishing.
[PLoS](http://www.plos.org) is leading the charge on presenting article
level metrics for all of its papers, but these remain papers that have
also been through a traditional peer review process. Very little that is
both radical with respect to the decision and means of publishing and
successful in getting traction amongst scientists has been seen as yet.

Out on the real web it has taken non-academics to demonstrate the truly
radical when it comes to publication. [](http://wikipedia.org) Whatever
you may think of the accuracy of [Wikipedia](http://wikipedia.org) in
your specific area, and I know it has some weaknesses in several of
mine, it is the first location that most people find, and the first
location that most people look for, when searching for factual
information on the web. [Roderic Page](http://iphylo.blogspot.com) put
up [some interesting
statistics](http://iphylo.blogspot.com/2009/09/google-wikipedia-and-eol.html)
when he looked this week at the top hits for over 5000 thousand mammal
names in Google. Wikipedia took the top spot 48% of the time and was in
the top 10 in virtually every case (97%). If you want to place factual
information on the web Wikipedia should be your first port of call.
Anything else is largely a waste of your time and effort. This doesn't
incidentally mean that other sources are not worthwhile or have a place,
but that people need to work with the assumption that people's first
landing point will be Wikipedia.

"But", I hear you say, "how do we know whether we can trust a given
Wikipedia article, or specific statements in it?"

The traditional answer has been to say you need to look in the logs,
check the discussion page, and click back the profiles of the people who
made specific edits. However this in inaccessible to many people, simply
because they do not know how to process the information. Very few
universities have an "Effective Use of Wikipedia 101" course. Mostly
because very few people would be able to teach it.

So I was very interested in an article on
[Mashable](http://mashable.com) about [marking up and colouring
Wikipedia text](http://mashable.com/2009/08/31/wikipedia-colors/)
according to its "trustworthiness". [Andrew
Su](http://friendfeed.com/asu) kindly pointed me in the direction of
[the group doing the
work](http://wikitrust.soe.ucsc.edu/index.php/Main_Page) and their
papers and presentations. The system they are using, which can be added
to any [MediaWiki](http://www.mediawiki.org/wiki/MediaWiki) installation
measures two things, how long a specific piece of text has stayed in
situ, and who either edited it, or left it in place. People who write
long lasting edits get higher status, and this in turn promotes the text
that they have "approved" by editing around but not changing.

This to me is very exciting because it provides extra value and
information for both users and editors without requiring anyone to do
any more work than install a plugin. The editors and writers simply
continue working as they have. The user can access an immediate view of
the trustworthiness of the article with a high level of granularity,
essentially at the level of single statements. And most importantly the
editor gets a metric, a number that is consistently calculated across
all editors, that they can put on a CV. Editors are peer reviewers, they
are doing review, on a constantly evolving and dynamic article that can
both change in response to the outside world and also be continuously
improved. Not only does the Wikipedia process capture most of the
valuable aspects of traditional peer review, it jettisons many of the
problems. But without some sort of reward it was always going to be
difficult to get professional scientists to be active editors. Trust
metrics could provide that reward.

Now there are many questions to ask about the calculation of this
"karma" metric, should it be subject biased so we know that highly
ranked editors have relevant expertise, or should it be general so as to
discourage highly ranked editors from modifying text that is outside of
their expertise? What should the mathematics behind it be? It will take
time clearly for such metrics to be respected as a scholarly
contribution, but equally I can see the ground shifting very rapidly
towards a situation where a lack of engagement, a lack of interest in
contributing to the publicly accessible store of knowledge, is seen as a
serious negative on a CV. However this particular initiative pans out it
is to me this is one of the first and most natural replacements for peer
review that could be effective within dynamic documents, solving most of
the central problems without requiring significant additional work.

I look forward to the day when I see CVs with a Wikipedia Karma Rank on
them. If you happen to be applying for a job with me in the future,
consider it a worthwhile thing to include.


I have long being sceptical of the costs and value delivered by our traditional methods of peer review. This is really on two fronts, firstly that the costs, where they have been estimated are extremely high, representing a multi-billion dollar subsidy by governments of the scholarly publishing industry.

Rogue Scholar Beiträge

A question of trust