Cameron Neylon

::: {.zemanta-img style="width: 250px;"}
<div>

[![Peer Review Monster](http://farm4.static.flickr.com/3112/3157622458_b19d039158_m.jpg "Peer Review Monster"){decoding="async"}](http://www.flickr.com/photos/33895652@N04/3157622458)
:   Image by [Gideon
    Burton](http://www.flickr.com/photos/33895652@N04/3157622458) via
    Flickr

</div>
:::

It hasn't been a real good week for peer review. In the same week that
[the Lancet](http://www.thelancet.com/ "The Lancet"){.zem_slink
rel="homepage"} fully [retract the original Wakefield MMR
article](http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(10)60175-7/fulltext)
(while keeping the retraction behind a login screen -- way to go there
on public understanding of science), the [main stream media went to
town](http://news.bbc.co.uk/2/hi/science/nature/8490291.stm) on the
report of [14 stem cell scientists writing an open
letter](http://eurostemcell.org/commentanalysis/peer-review) making the
claim that peer review in that area was being dominated by a small group
of people blocking the publication of innovative work. I don't have the
information to actually comment on the substance of either issue but I
do want to reflect on what this tells us about the state of peer review.

There remains much reverence of the traditional process of peer review.
I may be over interpreting the tenor of [Andrew Morrison's
editorial](http://www3.interscience.wiley.com/cgi-bin/fulltext/123248917/HTMLSTART)
in
[BioEssays](http://eu.wiley.com/WileyCDA/WileyTitle/productCd-BIES.html "BioEssays"){.zem_slink
rel="homepage"} but it seems to me that he is saying, as many others
have over the years "if we could just have the rigour of traditional
peer review with the ease of publication of the web then all our
problems would be solved". Â Scientists worship at the altar of peer
review, and I use that metaphor deliberately because it is rarely if
ever questioned. Somehow the process of peer review is supposed to
sprinkle some sort of magical dust over a text which makes it
"scientific" or "worthy", yet while we quibble over details of managing
the process, or complain that we don't get paid for it, rarely is the
fundamental basis on which we decide whether science is formally
published examined in detail.

There is a good reason for this. THE EMPEROR HAS NO CLOTHES! \[sorry,
had to get that off my chest\]. The evidence that peer review as
traditionally practiced is of any value at all is equivocal at best
([Science 214, 881;
1981](http://www.sciencemag.org/cgi/content/abstract/214/4523/881), [J
Clinical Epidemiology 50, 1189;
1998](http://jis.sagepub.com/cgi/content/abstract/30/1/2), [Brain 123,
1954;
2000](http://brain.oxfordjournals.org/cgi/content/full/123/9/1964),
[Learned Publishing 22, 117;
2009](http://clrcraldidcotgbr.library.ingentaconnect.com/content/alpsp/lp/2009/00000022/00000002/art00007)).
It's not even really negative. That would at least be useful. There are
a few studies that suggest peer review is somewhat better than throwing
a dice and a bunch that say it is much the same. It is at its best at
dealing with narrow technical questions, and at its worst at determining
"importance" is perhaps the best we might say. Which for anyone who has
tried to get published in a top journal or written a grant proposal
ought to be deeply troubling. Professional editorial decisions may in
fact be more reliable, something that Philip Campbell hints at in his
response to questions about the open letter \[[BBC
article](http://news.bbc.co.uk/1/hi/sci/tech/8490291.stm)\]:

> Our editors \[...\] have always used their own judgement in what we
> publish. We have not infrequently overruled two or even three
> sceptical referees and published a paper.

But there is perhaps an even more important procedural issue around peer
review. Whatever value it might have we largely throw away. Few journals
make referee's reports available, virtually none track the changes made
in response to referee's comments enabling a reader to make their own
judgement as to whether a paper was improved or made worse. Referees get
no public credit for good work, and no public opprobrium for poor or
even malicious work. And in most cases a paper rejected from one journal
starts completely afresh when submitted to a new journal, the work of
the previous referees simply thrown out of the window.

Much of the commentary around the open letter has suggested that the
peer review process should be made public. But only for published
papers. This goes nowhere near far enough. One of the key points where
we lose value is in the transfer from one journal to another. The
authors lose out because they've lost their priority date (in the worse
case giving the malicious referees the chance to get their paper in
first). The referees miss out because their work is rendered worthless.
Even the journals are losing an opportunity to demonstrate the high
standards they apply in terms of quality and rigor -- and indeed the
high expectations they have of their referees.

We never ask what the cost of not publishing a paper is or what the cost
of delaying publication could be. [Eric
Weinstein](http://www.eric-weinstein.net/) provides the most
sophisticated view of this that I have come across and I recommend
watching [his talk at Science in the 21st
Century](http://pirsa.org/08090036/) from a few years back. There is a
direct cost to rejecting papers, both in the time of referees and the
time of editors, as well as the time required for authors to reformat
and resubmit. But the bigger problem is the [opportunity
cost](http://en.wikipedia.org/wiki/Opportunity_cost "Opportunity cost"){.zem_slink
rel="wikipedia"} -- how much that might have been useful, or even
important, is never published? And how much is research held back by
delays in publication? How many follow up studies not done, how many
leads not followed up, and perhaps most importantly how many projects
not refunded, or only funded once the carefully built up expertise in
the form of research workers is lost?

Rejecting a paper is like gambling in a game where you can only win.
There are no real downside risks for either editors or referees in
rejecting papers. There are downsides, as described above, and those
carry real costs, but those are never borne by the people who make or
contribute to the decision. Its as though it were a futures market where
you can only lose if you go long, never if you go short on a stock. In
Eric's terminology those costs need to be carried, we need to require
that referees and editors who "go short" on a paper or grant are
required to unwind their position if they get it wrong. This is the only
way we can price in the downside risks into the process. If we want open
peer review, indeed if we want peer review in its traditional form,
along with the caveats, costs and problems, then the most important
advance would be to have it for unpublished papers.

Journals need to acknowledge the papers they've rejected, along with
dates of submission. Ideally all referees reports should be made public,
or at least re-usable by the authors. If full publication, of either the
submitted form of the paper or the referees report is not acceptable
then journals could publish a hash of the submitted document and reports
against a local key enabling the authors to demonstrate submission date
and the provenance of referees reports as they take them to another
journal.

In my view referees need to be held accountable for the quality of their
work. If we value this work we should also value and publicly laud good
examples. And conversely poor work should be criticised. Any scientist
has received reviews that are, if not malicious, then incompetent. And
even if we struggle to admit it to others we can usually tell the
difference between critical, but constructive (if sometimes brutal), and
nonsense. Most of us would even admit that we don't always do as good a
job as we would like. After all, why should we work hard at it? No
credit, no consequences, why would you bother? It might be argued that
if you put poor work in you can't expect good work back out when your
own papers and grants get refereed. This again may be true, but only in
the long run, and only if there are active and public pressures to raise
quality. None of which I have seen.

Traditional peer review is [hideously
expensive](http://www.rin.ac.uk/our-work/communicating-and-disseminating-research/activities-costs-and-funding-flows-scholarly-commu).
And currently there is little or no pressure on its contributors or
managers to provide good value for money. It is also unsustainable at
its current level. My solution to this is to radically cut the number of
peer reviewed papers probably by 90-95% leaving the rest to be published
as either pure data or pre-prints. But the whole industry is addicted to
traditional peer reviewed publications, from the funders who can't quite
figure out how else to measure research outputs, to the researchers and
their institutions who need them for promotion, to the publishers (both
OA and toll access) and metrics providers who both feed the addiction
and feed off it.

So that leaves those who hold the purse strings, the funders, with a
responsibility to pursue a value for money agenda. A good place to start
would be a serious critical analysis of the costs and benefits of peer
review.

*Addition after the fact: Pointed out in the comments that there are
other posts/papers I should have referred to where people* have *raised
similar ideas and issues. In particular [Martin Fenner's post at Nature
Network](http://network.nature.com/people/mfenner/blog/2009/07/13/the-value-of-peer-review).
The comments are particularly good as an expert analysis of the
usefulness of the kind of "value for money" critique I have made. Also a
[paper in the Arxiv](http://arxiv.org/abs/0911.0344) from Stefano
Allesina. Feel free to mention others and I will add them here.*

::: {.zemanta-pixie}
[![Reblog this post \[with
Zemanta\]](http://img.zemanta.com/reblog_e.png?x-id=b5743b7d-0c7c-4113-a3a2-10a6691b5e3f){.zemanta-pixie-img
decoding="async"}](http://reblog.zemanta.com/zemified/b5743b7d-0c7c-4113-a3a2-10a6691b5e3f/ "Reblog this post [with Zemanta]"){.zemanta-pixie-a}[]{.zem-script
.more-related .pretty-attribution}
:::


Image by Gideon Burton via Flickr  It hasn’t been a real good week for peer review.

Peer review: What is it good for?

Over the holidays I set up my own web presence both to be able to put more content and tools up and also to explore how I present myself as a scientist on the web. In a Semantic Web world where I am represented by a URL, what should that URL look like?

*[![Fireworks](http://farm2.static.flickr.com/1204/1264936124_6c28d80c36.jpg "Fireworks"){.alignleft
decoding="async" loading="lazy" width="210"
height="194"}](http://www.flickr.com/photos/mauricedb/1264936124/)Apologies
for any wierdness in your feed readers. The following is the reason why
as I try to get things working properly again.*

The past two years on this blog I wrote made some New Year's resolutions
and last year I assessed my performance against the previous year's
aims. This year I will admit to simply being a bit depressed about how
much I achieved in real terms and how effective I've been at getting
ideas out and projects off the ground. This year I want to do more in
terms of walking the walk, creating examples, or at least lashups of the
things I think are important.

One thing that has been going around in my head for at least 12 months
is the question of identity. How I control what I present, who I depend
on, and in the world of a semantic web where I am represented by a URL
what should actually be there when someone goes to that address. So the
positive thing I did over the holiday break, rather than write a new set
of resolutions was to start setting up my own presence on the web, to
think about what I might want to put there and what it might look like.

This process is not as far along as I would like but its far enough
along that this will be the last post at this
address.Â [OpenWetWare](http://openwetware.org "Open Wet Ware") has been
an amazing resource for me over the past several years and we will
continue to use the wiki for laboratory information and I hope to work
with the team in whatever way I can as the next generation of tools
develops. OpenWetWare was also a safe place where I could learn about
blogging without worrying about the mechanics, confident in the
knowledge that Bill Flanagan was covering the backstops. Bill is the
person who has kept things running through the various technical ups and
down and I'd particularly like to thank him for all his help.

However I have now learnt enough to be dangerous and want to try some
more things out on my own. More than can be conveniently managed on a
website that someone else has to look after. I will write a bit more
about the idea and choices I've made in setting up the site soon but for
the moment I just want to point you to the new site and offer you some
choices about subscribing to different feeds.

If you are on the feedburner feed for the blog you should be
automatically transferred over to the feed on the new site. If you're
reading in a feed reader you can check this by just clicking through to
the item on my site. If you end up at a url
startingÂ <http://cameronneylon.net/> then you are in the right place.
If not, just change your reader to point
atÂ <http://feeds.feedburner.com/ScienceInTheOpen>.

This feed will include posts on things like papers and presentations as
well as blog posts so if you are already getting that content in another
stream and prefer to just get the blog posts via RSS you should point
your reader atÂ <http://feeds.feedburner.com/ScienceInTheOpen_blog>. Â I
can't test this until I actually post something so just hold tight if it
doesn't work and I will try to get it working as soon as I can. The
comments feed for all seven of you subscribed to it should keep working.
All the posts are mirrored on the new site and will continue to be
available at OpenWetWare

Once again I'd like to thank all the people at OpenWetWare that got me
going in the blogging game and hope to see you over at the new site as I
figure out what it means to present yourself as a scientist on the web.

::: {.zemanta-pixie}
[![Reblog this post \[with
Zemanta\]](http://img.zemanta.com/reblog_e.png?x-id=e24cf0fc-08a4-45d6-b71f-49378a02cc99){.zemanta-pixie-img
decoding="async"}](http://reblog.zemanta.com/zemified/e24cf0fc-08a4-45d6-b71f-49378a02cc99/ "Reblog this post [with Zemanta]"){.zemanta-pixie-a}[]{.zem-script
.more-related .pretty-attribution}
:::


<em>
 Apologies for any wierdness in your feed readers. The following is the reason why as I try to get things working properly again.
</em>
The past two years on this blog I wrote made some New Year’s resolutions and last year I assessed my performance against the previous year’s aims.

Messaggi di Rogue Scholar

Peer review: What is it good for?

New Year – New me