Cameron Neylon

[Richard Grant](http://twitter.com/rpg7twit) asked me to record a few
minutes of audio talking about the ideas around the proposed [Workshop
on research metrics and
reputation](http://cameronneylon.net/blog/a-collaborative-proposal-on-research-metrics/)
for [The Scientist](http://www.the-scientist.com/). He did a very nice
job of writing some text around it so you should go and look at [the
piece over
atÂ NaturallyÂ Selected](http://blog.the-scientist.com/2010/10/25/what-can-we-do-for-you/)
at least partly because I seem to be unable to get the audio to embed
properly here...


Richard Grant asked me to record a few minutes of audio talking about the ideas around the proposed Workshop on research metrics and reputation for The Scientist.

Three minutes of audio &#8211; Hacking research reputation

::: {.zemanta-img}
<div>

[![The front and back of a UK cigarette packet (i\...](http://cameronneylon.net/wp-content/uploads/2010/09/300px-Cig.packet.750pix.jpg "The front and back of a UK cigarette packet (i..."){decoding="async"}](http://commons.wikipedia.org/wiki/File:Cig.packet.750pix.jpg)
:   Image via
    [Wikipedia](http://commons.wikipedia.org/wiki/File:Cig.packet.750pix.jpg)

</div>
:::

I had a bit of a rant at a [Science Online
London](http://www.scienceonlinelondon.org/) panel session on Saturday
with [Theo Bloom](http://twitter.com/theo_bloom), [Brian
Derby](http://www.materials.manchester.ac.uk/aboutus/staff/brianderby/),
and [Phil Lord](http://www.russet.org.uk/blog/) which people seemed to
like so it seemed worth repeating here. As usual when discussing
scientific publishing the dreaded issue of the Journal [Impact
Factor](http://en.wikipedia.org/wiki/Impact_factor "Impact factor"){.zem_slink
rel="wikipedia"} came up. While everyone complains about metrics I've
found that people in general seem remarkably passive when it comes to
challenging their use. Channeling [BjÃ¶rn
Brembs](http://bjoern.brembs.net/) [more than anything
else](http://bjoern.brembs.net/comment-n499.html) I said something
approximately like the following.

It seems bizarre that we are still having this discussion.
Thomson-Reuters say that the JIF shouldn't be used for judging
individual researchers, [Eugene
Garfield](http://en.wikipedia.org/wiki/Eugene_Garfield "Eugene Garfield"){.zem_slink
rel="wikipedia"}, the man who invented the JIF has consistently said it
should never be used to judge individual researchers. Even a cursory
look at the basic statistics should tell any half-competent scientist
with an ounce of quantitative analysis in their bones that the Impact
Factor of journals in which a given researcher publishes tells you
nothing whatsoever about the quality of their work.

Metrics are unlikely to go away -- after all, if we didn't have them we
might have to judge people's work by actually reading it -- but as
professional measurers and analysts of the world we should be
embarrassed to use JIFs to measure people and papers. It is quite simply
bad science. It is also bad management. If our managers and leaders have
neither the competence nor the integrity to use appropriate measurement
tools then they should be shamed into doing so. If your managers are not
competent to judge the quality of your work without leaning on spurious
measures your job and future is in seriousÂ jeopardy. But more
seriously, if as professional researchers we don't have the integrity to
challenge the fundamental methodological flaws in using JIFs to judge
people and the appalling distortion of scientific communication that
this creates then I question whether our own research methodology can be
trusted either.

My personal belief is that we should be focussing on developing
effective and diverse measures of the [re-use of research
outputs](http://cameronneylon.net/blog/metrics-of-use-how-to-align-researcher-incentives-with-outcomes/).
By measuring *use* rather than merely prestige we can go much of the way
of delivering on the so-called impact agenda, optimising our use of
public funds to generate outcomes but while retaining some say over the
types of outcomes that are important and what timeframes they are
measured over. But whether or not you agree with my views it seems to me
critical that we, as hopefully competent scientists, at least debate
what it is we are trying to optimise and what are the appropriate things
we should be trying to measure so we can work on providing reliable and
sensible ways of doing that.

::: {.zemanta-pixie}
[![Enhanced by
Zemanta](http://img.zemanta.com/zemified_e.png?x-id=d5fca92e-6607-447d-8e90-c6756b682d4a){.zemanta-pixie-img
decoding="async"}](http://www.zemanta.com/ "Enhanced by Zemanta"){.zemanta-pixie-a}[]{.zem-script
.more-related .pretty-attribution .paragraph-reblog}
:::


Image via Wikipedia  I had a bit of a rant at a Science Online London panel session on Saturday with Theo Bloom, Brian Derby, and Phil Lord which people seemed to like so it seemed worth repeating here. As usual when discussing scientific publishing the dreaded issue of the Journal Impact Factor came up. While everyone complains about metrics I’ve found that people in general seem remarkably passive when it comes to challenging their use.

Warning: Misusing the journal impact factor can damage your science!

Cultural change won’t happen overnight and may never happen at all. Policy mandates are a blunt instrument and may lead to a backlash. So how do we align incentives for researchers with the aim of making their research outputs more widely and effectively useable? Get the metrics right and the competitive market could manage the rest of the optimisation problem for us.

::: {.zemanta-img}
<div>

[![slices of carrot](http://cameronneylon.net/wp-content/uploads/2010/06/300px-Slices_of_carrot.jpg "slices of carrot"){decoding="async"}](http://commons.wikipedia.org/wiki/File:Slices_of_carrot.jpg)
:   Image via
    [Wikipedia](http://commons.wikipedia.org/wiki/File:Slices_of_carrot.jpg)

</div>
:::

It has become reflexive in the Open Communities to talk about a need for
"cultural change". The obvious next step becomes to find strong and
widely respected advocates of change, to evangelise to young
researchers, and to hope for change to follow. Inevitably this process
is slow, perhaps so slow as to be ineffective. So beyond the grassroots
evangelism we move towards policy change as a top down mechanism for
driving improved behaviour. If funders demand that data be open, that
papers be accessible to the wider community, as a condition of funding
then this will happen. The [NIH mandate](http://publicaccess.nih.gov/)
and the work of the [Wellcome
Trust](http://www.wellcome.ac.uk/ "Wellcome Trust"){.zem_slink
rel="homepage"} on Open Access show that this can work, and indeed that
mandates in some form are necessary to raise levels of compliance to
acceptable levels.

But policy is a blunt instrument, and researchers being who they are
don't like to be pushed around. Passive aggressive responses from
researchers are relatively ineffectual in the peer reviewed articles
space. A paper is a paper. If its under the right licence then things
will probably be ok and a specific licence is easy to mandate. Data
though is a different fish. It is very easy to comply with a data
availability mandate but provide that data in a form which is totally
useless. Indeed it is rather hard work to provide it in a form that is
useful. Data, software, reagents and materials, are incredibly diverse
and it is difficult to make good policy that can be both effective and
specific enough, as well as general enough to be useful. So beyond the
policy mandate stick, which will only ever provide a minimum level of
compliance, how do we motivate researchers to putting the effort into
making their outputs available in a *useful* form? How do we encourage
them to *want* to do the right thing?Â After all what we want to enable
is re-use.

We need more sophisticated motivators than blunt policy instruments, so
we arrive at metrics. Measuring the ouputs of researchers. There has
been a wonderful [animation](http://www.youtube.com/watch?v=u6XAPnuFjJc)
illustrating a [Daniel Pink](http://www.danpink.com/) talk doing the
rounds in the past week. Well worth a look and important stuff but I
think a naive application of it to researchers' motivations would miss
two important aspects. Firstly, money is never "off the table" in
research. We are always to some extent limited by resources. Secondly
the intrinsic motivators, the internal metrics that matter to
researchers, are tightly tied to the metrics that are valued by their
communities. In turn those metrics are tightly tied to resource
allocation. Most researchers value their papers, the places they are
published and the citations received, as measures of their value,
because that's what their community values. The system is highly
leveraged towards rapid change, if and only if a research *community*
starts to value a different set of metrics.

What might the metrics we would like to see look like? I would suggest
that they should focus on what we want to see happen. We want return on
the public investment, we want value for money, but above all we want to
maximise the opportunity for research outputs to be used and to be
useful. We want to optimise the usability and re-usability of research
outputs and we want to encourage researchers to do that optimisation.
Thus if our metrics are metrics of *use* we can drive behaviour in the
right direction.

If we optimise for re-use then we automatically value access, and we
automatically value the right licensing arrangements (or lack thereof).
If we value and measure use then we optimise for the release of data in
useful forms and for the release of open source research software. If we
optimise for re-use, for discoverability, and for value add, then we can
automatically tension the loss of access inherent in publishing in
*Nature* or *Science* vs the enhanced discoverability and editorial
contribution and put a real value on these aspects. We would stop
arguing about whether tenure committees should value blogging and start
asking how much those blogs were used by others to provide outreach,
education, and research outcomes.

For this to work there would need to be mechanisms that automatically
credit the use of a much wider range of outputs. We would need to cite
software and data, would need to acknowledge the providers of metadata
that enabled our search terms to find the right thing, and we would need
to aggregate this information in a credible and transparent way. This is
technically challenging, and technically interesting, but do-able. Many
of the pieces are in place, and many of the community norms around
giving credit and appropriate citation are in place, we're just not too
sure how to do it in many cases.

Equally this is a step back towards what the mother of all metrics, the
[Impact
Factor](http://en.wikipedia.org/wiki/Impact_factor "Impact factor"){.zem_slink
rel="wikipedia"} was originally about. The IF was intended as a way of
measuring the *use* of journals through counting citations, as a means
of helping librarians to choose which journals to subscribe to. [Article
Level
Metrics](http://www.plosbiology.org/article/info:doi/10.1371/journal.pbio.1000242)
are in many ways the obvious return to this where we want to measure the
outputs of specific researchers. The
[H-factor](http://en.wikipedia.org/wiki/H-index) for all its weaknesses
is a measure of re-use of outputs through formal citations. Influence
and impact are already an important motivator at the policy level.
Measuring use is actually a quite natural way to proceed. If we can get
it right it might also provide the motivation we want to align
researcher interests with the wider community and optimise access to
research for both researchers and the public.

###### Related articles by Zemanta {#related-articles-by-zemanta .zemanta-related-title}

- [John Wilbanks on Science Commons, and generativity in
  science](http://www.ethanzuckerman.com/blog/2010/03/09/john-wilbanks-on-science-commons-and-generativity-in-science/)
  (ethanzuckerman.com)

::: {.zemanta-pixie}
[![Reblog this post \[with
Zemanta\]](http://img.zemanta.com/reblog_e.png?x-id=d8fea424-9fd4-40df-8e52-00e3e46a388b){.zemanta-pixie-img
decoding="async"}](http://reblog.zemanta.com/zemified/d8fea424-9fd4-40df-8e52-00e3e46a388b/ "Reblog this post [with Zemanta]"){.zemanta-pixie-a}[]{.zem-script
.more-related .pretty-attribution .paragraph-reblog}
:::


Image via Wikipedia  It has become reflexive in the Open Communities to talk about a need for “cultural change”. The obvious next step becomes to find strong and widely respected advocates of change, to evangelise to young researchers, and to hope for change to follow. Inevitably this process is slow, perhaps so slow as to be ineffective.

Messages de Rogue Scholar

Three minutes of audio – Hacking research reputation

Warning: Misusing the journal impact factor can damage your science!

Metrics of use: How to align researcher incentives with outcomes