Cameron Neylon

Late last year the [Research Information Network](http://www.rin.ac.uk/)
held a [workshop](http://www.rin.ac.uk/what-cost) in London to launch a
report, and in many ways more importantly, a detailed economic model of
the scholarly publishing industry. The model aims to capture the
diversity of the scholarly publishing industry and to isolate costs and
approaches to enable the user to ask questions such as "what is the
consequence of moving to a 95% author pays model" as well as to simply
ask how much money is going in and where it ends up. I've been meaning
to write about this for ages but a couple of things in the last week
have prompted me to get on and do it.

The first of these was an announcement by email \[can't find a copy
online at the moment\] by the [EPSRC](http://www.epsrc.ac.uk), the UK's
main funder of physical sciences and engineering. While the requirement
for a two page enconomic impact statement for each grant proposal [got
more
headlines](http://www.timeshighereducation.co.uk/story.asp?storycode=405350),
what struck me as much more important were two other policy changes. The
first was that, unless specifically invited, rejected proposals *can not
be resubmitted*. This may seem strange, particularly to US researchers,
where a process of refinement and resubmission, perhaps multiple times,
is standard, but the [BBSRC](http://www.bbsrc.ac.uk) (UK biological
sciences funder) has had a similar policy for some years. The second,
frankly somewhat scarey change, is that some proportion of researchers
that have a history of rejection will be barred from applying
altogether. What is the reason for these changes? Fundamentally the
burden of carrying out peer review on all of the submitted proposals is
becoming too great.

The second thing was that, for the first time, I have been involved in
refereeing a paper for a [Nature Publishing Group](http://nature.com)
journal. Now I like to think, like I guess everyone else does, that I do
a reasonable job of paper refereeing. I wrote perhaps one and a half
sides of A4 describing what I thought was important about the paper and
making some specific criticisms and suggestions for changes. The paper
went around the loop and on the second revision I saw what the other
referees had written; pages upon pages of closely argued and detailed
points. Now the other referees were much more critical of the paper but
nonetheless this supported a suspicion that I have had for some time,
that refereeing at some high impact journals is qualitatively different
to what the majority of us receive, and probably deliver; an often form
driven exercise with a couple of lines of comments and complaints. This
level of quality peer review takes an awful lot of time and it costs
money; money that is coming from somewhere. Nonetheless it provides
better feedback for authors and no doubt means the end product is better
than it would otherwise have been.

The final factor was a [blog
post](http://molphil.wordpress.com/2009/02/27/socialism-in-science-or-why-open-access-may-ultimately-fail/)
from [Molecular Philosophy](http://molphil.wordpress.com) discussing why
the author felt Open Access Publishers are, if not doomed to failure,
then face a very challenging road ahead. The centre of the argument as I
understand it focused around the costs of high impact journals,
particularly the costs of selection, refinement, and preparation for
print. Broadly speaking I think it is generally accepted that a volume
model of OA publication, such as that practiced by [PLoS
ONE](http://www.plosone.org/) and [BMC](http://www.biomedcentral.com)
can be profitable. I think it is also generally accepted that a
profitable business model for high impact OA publication has yet to be
convincingly demonstrated. The question I would like to ask though is
different. The Molecular Philosophy post skips the zeroth order
questions. Can we *afford* high impact publications?

Returning to the RIN funded study and model of scholarly publishing some
very interesting points came out \[see [Daniel Hull's
presentation](http://www.rin.ac.uk/files/Daniel_Hulls_What_cost_11Dec08.pdf)
for most of the data here\]. The first of these, which in retrospect is
obvious but important, is that the vast majority of the costs of
producing a paper are incurred in doing the research it describes
(Â£116G worldwide). The second biggest contributor? Researchers reading
the papers (Â£34G worldwide). Only about 14% of the costs of the total
life cycle are actually taken up with costs directly attributable to
publication. But that is the 14% we are interested in, so how does it
divide up?

The "Scholarly Communication Process" as everything in the middle is
termed in the model is divided up into actual publication/distribution
costs (Â£6.4G), access provision costs (providing libraries and internet
access, Â£2.1G) and the costs of researchers looking for articles
(Â£16.4G). Yes, the biggest cost is the time you spend trying to find
those papers. Arguably that is a sunk cost in as much as once you've
decided to do research searching for information is a given, but it does
make the point that more efficient searching has the potential to save a
lot of money. In any case it is a non-cash cost in terms of journal
subscriptions or author charges.

So to find the real costs of publication *per se* we need to look inside
that Â£6.4. Of the costs of actually publishing the articles the biggest
single cost is peer review weighing in at around Â£1.9G globally, just
ahead of fixed "first copy" publication costs of Â£1.8G. So 29% of the
total costs incurred in publication and distribution of scholarly
articles arises from the cost of peer review.

There are lots of other interesting points in the reports and models
(the UK is a net exporter of peer review, but the UK publishes more
articles than would be expected based on its subscription expenditure)
but the most interesting aspect of the model is its ability to model
changes in the publishing landscape. The first scenario presented is one
in which publication moves to being 90% electronic. This actually leads
to a fairly modest decrease in costs overall with a total overall saving
of a little under Â£1G (less than 1%). Modeling a move to a 90% author
pays model (assuming 90% electronic only) leads to very little change
overall, but interestingly that depends significantly on the cost of
systems put in place to make author payments. If these are expensive and
bureaucratic then the costs can rise as many small payments are more
expensive than few big ones. But overall the costs shouldn't need to
change much, meaning if mechanisms can be put in place to move the money
around, the business models should ultimately be able to make sense.
None of this however helps in figuring out how to manage a *transition*
from one system to another, when for all useful purposes costs are
likely to double in the short term as systems are duplicated.

The most interesting scenario, though was the third. What happens as
research expands. A 2.5% real increase year on year for ten years was
modeled. This may seem profligate in today's economic situation but with
many countries explicitly spending stimulus money on research, or
already engaged in large scale increases of structural research funding
it may not be far off. This results in 28% more articles, 11% more
journals, a 12% increase in subscription costs (assuming of course that
only the real cost increases are passed on) and a 25% increase in the
costs of peer review (Â£531M on a base of Â£1.8G).

I started this post talking about proposal refereeing. The increased
cost in refereeing proposals as the volume of science increases would be
added on top of that for journals. I think it is safe to say that the
increase in cost would be of the same order. The refereeing system is
already struggling under the burden. Funding bodies are creating new,
and arguably totally unfair, rules to try and reduce the burden,
journals are struggling to find referees for paper. Increases in the
volume of science, whether they come from increased funding in the
western world or from growing, increasingly technology driven, economies
could easily increase that burden by 20-30% in the next ten years. I am
sceptical that the system, as it currently exists, can cope and I am
sceptical that peer review, in its current form is affordable in the
medium to long term.

So, bearing in mind [Paulo's
admonishment](http://blindscientist.genedrift.org/2009/01/22/why-open-scienceresearch-won%E2%80%99t-work-at-least-for-now-a-story-from-friendfeed/)
that I need to offer solutions as well as problems, what can we do about
this? We need to find a way of doing peer review effectively, but it
needs to be more efficient. Equally if there are areas where we can save
money we should be doing that. Remember that Â£16.4G just to find the
papers to read? I believe in post-publication peer review because it
reduces the costs and time wasted in bringing work to community view and
because it makes the filtering and quality assurance of that published
work continuous and ongoing. But in the current context it offers
significant cost savings. A significant proportion of published papers
are never cited. To me it follows from this that there is no point in
peer reviewing them. Indeed citation is an act of post-publication peer
review in its own right and it has recently been shown that Google
[PageRank type algorithms do a pretty good job of identifying important
papers](http://arxiv.org/abs/0901.2640) without any human involvement at
all (beyond the act of citation). Of course for PageRank mechanisms to
work well the citation and its full context are needed making OA a
pre-requisite.

If refereeing can be restricted to those papers that are worth the
effort then it should be possible to reduce the burden significantly.
But what does this mean for high impact journals? The whole point of
high impact journals is that they are hard to get into. This is why both
the editorial staff and peer review costs are so high for them. Many
people make the case that they are crucial for helping to filter out the
important papers (remember that Â£16.4G again). In turn I would argue
that they reduce value by making the process of deciding what is
"important" a closed shop, taking that decision away, to a certain
extent, from the community where I feel it belongs. But at the end of
the day it is a purely economic argument. What is the overall cost of
running, supporting through peer review, and paying for, either by
subscription or *via* author charges, a journal at the very top level?
What are the benefits gained in terms of filtering and how do they
compare to other filtering systems. Do the benefits justify the costs?

If we believe that there are better filtering systems possible, then
they need to be built, and the cost benefit analysis done. The
opportunity is coming soon to offer different, and more efficient,
approaches as the burden becomes too much to handle. We either have to
bear the cost or find better solutions.

*\[This has got far too long already -- and I don't have any simple
answers in terms of refereeing grant proposals but will try to put some
ideas in another post which is long overdue in response to [a
promise](http://duncan.hull.name/2008/08/14/anyone-for-a-game-of-fantasy-science-funding/)
to [Duncan Hull](http://duncan.hull.name)\]*


Late last year the Research Information Network held a workshop in London to launch a report, and in many ways more importantly, a detailed economic model of the scholarly publishing industry.

What is the cost of peer review? Can we afford (not to have) high impact journals?

The [Research Information Network](http://www.rin.ac.uk/) has put out a
cal for expressions of interest in running a research project on how Web
2.0 tools are changing scientific practice. The project will be funded
up to Â£90,000. Expressions of interest are due on Monday 3 November
(yes next week) and the projects are due to start in January. You can
see the [call in full
here](http://dl.getdropbox.com/u/24359/RIN_half_page_311.pdf) but in
outline RIN seeking evidence whether web 2.0 tools are:

â€¢ making data easier to share, verify and re-use, or otherwise

facilitating more open scientific practices;

â€¢ changing discovery techniques or enhancing the accessibility of

research information;

â€¢ changing researchersâ€™ publication and dissemination behaviour,

(for example, due to the ease of publishing work-in-progress and

grey literature);

â€¢ changing practices around communicating research findings (for

example through opportunities for iterative processes of feedback,

pre-publishing, or post-publication peer review).

Now we as a community know that there are cases where all of these are
occurring and have fairly extensively documented examples. The question
is obviously one of the degree of penetration. Again we know this is
small â€" Iâ€™m not exactly sure how you would quantify it.

My challenge to you is whether it would be possible to use the tools and
community we already have in place to carry out the project? In the past
weâ€™ve talked a lot about aggregating project teams and distributed
work but the problem has always been that people donâ€™t have the time
to spare. We would need to get some help from social scientists on
process and design of the investigation but with Â£90,000 there is
easily enough money to pay people properly for their time. Indeed I know
there are some people out there freelancing already who are in many ways
already working on these issues anyway. So my question is: Are people
interested in pursuing this? And if so, what do you think your hourly
rate is?


The Research Information Network has put out a cal for expressions of interest in running a research project on how Web 2.0 tools are changing scientific practice. The project will be funded up to Â£90,000. Expressions of interest are due on Monday 3 November (yes next week) and the projects are due to start in January.

Rogue Scholar Beiträge

What is the cost of peer review? Can we afford (not to have) high impact journals?

Call for submissions for a project on The Use and Relevance of Web 2.0 Tools for Researchers