Björn Brembs

The other day I was
[alerted](https://curt-rice.com/2014/01/24/the-shocking-truth-about-the-quality-of-research-its-not-getting-better-2/)
to an interesting evaluation of international citation data. The author,
Curt Rice, mentions a particular aspect of the data:

> In 2000, 25% of Norwegian articles remained uncited in their first
> four years of life. By 2009, this had fallen to about 15%. This shows
> that the "bottom" isn't pulling the average down. In fact, it's
> raising it, making more room for the top to pull us even higher.

The context here is that the "bottom" refers to scientific articles that
aren't cited, assuming that no citations mean low scientific quality of
that article. Leaving aside the very tentative and hotly debated
connection between citations and 'quality' (whatever that actually
means\*) in general, let's look at just the specific rationale that a
decrease in the fraction of articles that go uncited should indicate an
increase in the overall quality of research.

To make my case, I'll take the perspective of someone who strongly
believes that the rejection rate of a journal is indicative of that
journal's 'quality' (i.e, a high rejection rate makes sure that only the
[world's best
science](https://www.nature.com/citation_pages/images/nature_com.gif) is
being published). From that perspective, a decreasing fraction of papers
remaining uncited is just as bad for science as decreasing rejection
rates: surely, just as not every paper can be good enough to deserve to
be published, even fewer would be good enough to deserve to be cited? An
increasing number of articles with any citations at all can thus only
mean one thing: the [Dunning-Kruger
Effect](https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect) has
come to science. We have now let so many incompetent people join the
scientific enterprise, that they cannot discern between good science and
bad science any more and cite even the worst bottom of the scientific
literature, as if it was even worth paying attention to. As a
consequence of the rise of these
[bulk-publishing](https://www.nature.com/news/2008/080702/full/454011a.html)
new journals which flood the scholarly literature with crap, articles
get published who would never have gotten published before and in their
authors' unwitting incompetence, they cite each other out of sheer
ignorance. With [80% of all submitted manuscripts being
junk](https://eagereyes.org/series/peer-review/peer-review-part-5-the-importance-of-gatekeepers),
clearing what passes for peer-review these days ceases to be an
indicator of quality and with almost all papers being cited, citations
has become useless, too.

This may sound outrageous to people who visit this obscure blog, but if
you follow the links in the paragraph above, you'll find precisely this
condescension and arrogance.

 

\* Obviously, a more easily defensible relation were that between
citations and utility


The other day I was alerted to an interesting evaluation of international citation data. The author, Curt Rice, mentions a particular aspect of the data:  The context here is that the “bottom” refers to scientific articles that aren’t cited, assuming that no citations mean low scientific quality of that article.

Citation inflation and incompetent scientists

University of Glasgow

Roderic Page

![359f89198dca80b0f99b3208e1cfedde](http://lh6.ggpht.com/-9BAecOq8guY/UZc2JsdUYBI/AAAAAAAACyM/vpOij2hvcpU/359f89198dca80b0f99b3208e1cfedde.jpeg?imgmax=800 "359f89198dca80b0f99b3208e1cfedde.jpeg"){border="0"
width="128" height="128" style="float:right;"}

> iPhylo: GBIF specimens in BioStor: who are the top ten
> museums?(Unfortunately not @[amnh](https://twitter.com/amnh))
> [iphylo.blogspot.com/2012/02/gbif-s...](http://t.co/umhLz8MR8d "http://iphylo.blogspot.com/2012/02/gbif-specimens-in-biostor-who-are-top.html?spref=tw")
>
> --- Susan Perkins (@NYCuratrix) [May 14,
> 2013](https://twitter.com/NYCuratrix/status/334356241226821636)

  
Ideas on measuring the \"impact\" of a natural history collection have
been bubbling along, as reflected in [recent comments on
iPhylo](http://iphylo.blogspot.co.uk/2012/02/linking-gbif-and-biodiversity-heritage.html#comment-898170035),
and some offline discussions I\'ve been having with David Blackburn and
Alan Resetar.  
  
My focus has been at the specimen-level, with a view to motivation the
adoption of persistent specimen-level identifiers so that we can
citations of specimens over time (e.g., in publications and databases
such as GenBank). Not only does this provide a measure of the \"impact\"
of a collection, it helps with provenance. If we sequence a specimen
that is subsequently assigend to a different taxon and we have a way of
tracking that specimen via its identifier, then we can transmit that new
identification to other consumers of data based on that specimen. For
example, we could automatically notify GenBank that what we thought was
an *x* is actually a *y*.  
  
So I made a simple [\"league table\" of museum
collections](http://iphylo.blogspot.co.uk/2012/02/gbif-specimens-in-biostor-who-are-top.html)
based on specimens cited in [BioStor](http://biostor.org). There are all
sorts of issues with this approach. Once you rank collections, people
may use that to argue some can be axed and more resources funnelled into
others. A more positive approach would be to indetify collections that
are underused, and try and figure out why. And in the same way that
taxonomic papers may have a citation long life, specimens may sit in a
museum for a long time before being cited (for example, when eventually
recognised as a new species
[doi:10.1016/j.cub.2012.10.029](http://dx.doi.org/10.1016/j.cub.2012.10.029)).
So, metrics can be a double-edged sword.  
  
Citing specimens is a useful metric, but not all citations are equal,
and not all citations are immediate. A specimen that yields DNA
sequences that are published in, say, *Nature*, arguably has more weight
than a specimen listed in a rarely cited paper. Likewise, subsequent
citations of a paper that cites a specimen should confer more weight on
the value of that specimen. Elsewhere
([doi:10.1093/bib/bbn022](http://dx.doi.org/10.1093/bib/bbn022),
preprint here:
[hdl:10101/npre.2008.1760.1](http://hdl.handle.net/10101/npre.2008.1760.1))
I\'ve argued for a Google PageRank-style way to measure the impact of a
specimen that takes into account papers and other objects derived from a
specimen (e.g., images, sequences).  
  
Meanwhile, Morgan Jackson alerted me to a quicker way to get a measure
of the impact of the collection.  

> @[rdmpage](https://twitter.com/rdmpage)
> @[nycuratrix](https://twitter.com/nycuratrix) Check Nature for a
> recent note about this. A bird museum calc. their collection\'s
> h-score from papers citing specimens
>
> --- Morgan Jackson (@BioInFocus) [May 16,
> 2013](https://twitter.com/BioInFocus/status/335007197568311298)

::: {.altmetric-embed style="float:right;" badge-type="donut" doi="10.1038/493480b"}
:::

  
The \"short note\" Morgan refers to is by [Kevin
Winker](http://www.kevinwinker.org/) and Jack J. Withrow:  

> Winker, K., & Withrow, J. J. (2013). Natural history: Small
> collections make a big impact. Nature, 493(7433), 480--480.
> [doi:10.1038/493480b](http://dx.doi.org/10.1038/493480b)

  
They constructed a [Google Scholar
profile](http://scholar.google.com/citations?user=EbOuSfwAAAAJ) and
collected papers that cite the University of Alaska Museum\'s bird
collection (see [here for full
details](http://www.kevinwinker.org/Creation%20of%20UAM%20Birds%20Google%20Scholar%20profile.pdf)).
The *h*-score of this collection of papers is 42, which Winkler and
Withrow note is \"equivalent to an average Nobel laureate in physics\".
Here\'s the graph of citations over time:  
  
![Chart
1](http://lh5.ggpht.com/-UqMqnwh2uEA/UZTgOE84odI/AAAAAAAACx8/1bbwEVOpJlI/chart%252520%2525281%252529.png?imgmax=800 "chart (1).png"){style="display:block; margin-left:auto; margin-right:auto;"
border="0" width="475" height="90"}  
It\'s a neat trick, if a little time consuming. But one advantage it has
is that it puts collections on a similar footing to individual
researchers. You could imagine asking the question \"how much money
would you spend supporting a researcher at this level?\" How does this
compare to the resources actually being spent?  
  
One thing I hope will emerge from discussions like this is a desire to
make specimens first-class citizens of the web, with stable identifiers
that enable them to be cited in the same way we cite papers and,
increasingly, data sets.


Ideas on measuring the "impact" of a natural history collection have been bubbling along, as reflected in recent comments on iPhylo, and some offline discussions I've been having with David Blackburn and Alan Resetar.

The impact of museum collections: one collection ≈ one Nobel Prize

This is an edited version of the text that I spoke from at the Altmetrics Workshop in Koblenz in June. Impact as re-use and the way it enables us to reframe the argument around the impact and dissemination of curiosity driven research.

Cameron Neylon

::: {.zemanta-img}
::: {.wp-caption .alignleft style="width: 250px"}
[![Open](http://cameronneylon.net/wp-content/uploads/2011/07/185151669_b9cb263130_m10.jpg "Open"){decoding="async"
loading="lazy" width="240"
height="180"}](http://www.flickr.com/photos/18883956@N00/185151669)

Image by tribalicious via Flickr
:::
:::

*This is an edited version of the text that I spoke from at the
[Altmetrics Workshop](http://altmetrics.org/workshop2011/) in
[Koblenz](http://en.wikipedia.org/wiki/Koblenz "Koblenz"){.zem_slink
rel="wikipedia"} in June. There is also an [audio recording of the
talk](http://www.archive.org/details/Re-useAsImpactTalkAtaltmetricsWorkshop)
I gave available as well as the [submitted
abstract](http://altmetrics.org/workshop2011/neylon-v0/) for the
workshop.*

I developed an interest in research evaluation as an advocate of open
research process. It is clear that researchers are not going to change
themselves so someone is going to have to change them and it is funders
who wield the biggest stick. The only question, I thought, Â was how to
persuade them to use it

Of course itâ€™s not that simple. It turns out that funders are highly
constrained as well. They can lead from the front but not too far out in
front if they want to retain the confidence of their community. And the
actual decision making processes remain dominated by senior researchers.
Successful senior researchers with little interest in rocking the boat
too much.

The thing you realize as you dig deeper into this as that the key lies
in finding motivations that work across the interests of different
stakeholders. The challenge lies in finding the shared objectives. What
it is that unites both researchers and funders, as well as government
and the wider community. So what can we find that is shared?

Iâ€™d like to suggest that one answer to that is Impact. The research
community as a whole has stake in convincing government that research
funding is well invested. Government also has a stake in understanding
how to maximize the return on its investment. Researchers do want to
make a difference, even if that difference is a long way off. You need a
scattergun approach to get the big results, but that means supporting a
diverse range of research in the knowledge that some of it will go
nowhere but some of it will pay off.

Impact has a bad name but if we step aside from the gut reactions and
look at what we actually want out of research then we start to see a
need to raise some challenging questions. What is research for?Â  What
is its role in our society really? What outcomes would we like to see
from it, and over what timeframes? What would we want to evaluate those
outcomes against? Economic impact yes, as well as social, health,
policy, and environmental impact. This is called the 'triple bottom
line' in Australia. But alongside these there is also research impact.

All these have something in common. Re-use. What we mean by impact is
re-use. Re-use in industry, re-use in public health and education,
re-use in policy development and enactment, and re-use in research.

And this frame brings some interesting possibilities. We can measure
some types of re-use. Citation, retweets, re-use of data or materials,
or methods or software. We can think about gathering evidence of other
types of re-use, and of improving the systems that acknowledge re-use.
If we can expand the culture of citation and linking to new objects and
new forms of re-use, particularly for objects on the web, where there is
some good low hanging fruit, then we can gather a much stronger and more
comprehensive evidence base to support all sorts of decision making.

There are also problems and challenges. The same ones that any social
metrics bring. Concentration and community effects, the Matthew effect
of the rich getting richer. We need to understand these feedback effects
much better and I am very glad there are significant projects addressing
this.

But there is also something more compelling for me in this view. It
let's us reframe the debate around basic research. The argument goes we
need basic research to support future breakthroughs. We know neither
what we will need nor where it will come from. But we know that its very
hard to predict -- that's why we support curiosity driven research as an
important part of the portfolio of projects. Yet the dissemination of
this investment in the future is amongst the weakest in our research
portfolio. At best a few papers are released then hidden in journals
that most of the world has no access to and in many cases without the
data, or other products either being indexed or even made available. And
this lack of effective dissemination is often *because* the work is
perceived as low, or perhaps better, *slow* impact.

We may not be able to demonstrate or to measure significant re-use of
the outputs of this research for many years. But what we can do is focus
on optimizing the capacity, the potential, for future exploitation.
Where we can't demonstrate re-use and impact we should demand that
researchers demonstrate that they have optimized their outputs to enable
future re-use and impact.

And this brings me full circle. My belief is that the way to ensure the
best opportunities for downstream re-use, over all timeframes, is that
the research outputs are open, in the [Budapest
Declaration](http://www.soros.org/openaccess/read.shtml) sense. But we
don't have to take my word for it, we can gather evidence. Making
everything naively open will not always be the best answer, but we need
to understand where that is and how best to deal with it. We need to
gather evidence of re-use over time to understand how to optimize our
outputs to maximize their impact.

But if we choose to value re-use, to value the downstream impact that
our research or have, or could have, then we can make this debate not
about politics or ideology but how about how best to take the public
investment in research and to invest it for the outcomes that we need as
a society.

 

 

 

 

###### Related articles {#related-articles .zemanta-related-title}

- [Evidence to the European Commission Hearing on Access to Scientific
  Information](http://cameronneylon.net/presentations/evidence-to-the-european-commission-hearing-on-access-to-scientific-information/)
  (cameronneylon.net)
- [QUT \| Library \| Measuring research
  impact](http://www.library.qut.edu.au/services/research/impact.jsp)
  (library.qut.edu.au)
- [iASSIST: Data Reuse
  Modelling](http://jwhyteappleby.wordpress.com/2011/06/04/iassist-data-reuse-modelling/)
  (jwhyteappleby.wordpress.com)

::: {.zemanta-pixie}
[![Enhanced by
Zemanta](http://img.zemanta.com/zemified_e.png?x-id=621c3c65-57b4-41a6-9849-f60f4f5ee215){.zemanta-pixie-img
decoding="async"}](http://www.zemanta.com/ "Enhanced by Zemanta"){.zemanta-pixie-a}
:::


Image by tribalicious via Flickr
<em>
 This is an edited version of the text that I spoke from at the Altmetrics Workshop in Koblenz in June. There is also an audio recording of the talk I gave available as well as the submitted abstract for the workshop.
</em>
I developed an interest in research evaluation as an advocate of open research process.

Messaggi di Rogue Scholar

Citation inflation and incompetent scientists

The impact of museum collections: one collection ≈ one Nobel Prize

(S)low impact research and the importance of open in maximising re-use