Olivia Aguiar

<figure class="wp-block-image size-large">
<img
src="https://www.scholcommlab.ca/wp-content/uploads/2022/05/MetricsLiteracies-1024x858.jpeg"
class="wp-image-4038" loading="lazy" decoding="async"
srcset="https://www.scholcommlab.ca/wp-content/uploads/2022/05/MetricsLiteracies-1024x858.jpeg 1024w, https://www.scholcommlab.ca/wp-content/uploads/2022/05/MetricsLiteracies-300x251.jpeg 300w, https://www.scholcommlab.ca/wp-content/uploads/2022/05/MetricsLiteracies-768x644.jpeg 768w, https://www.scholcommlab.ca/wp-content/uploads/2022/05/MetricsLiteracies-1536x1287.jpeg 1536w, https://www.scholcommlab.ca/wp-content/uploads/2022/05/MetricsLiteracies-2048x1716.jpeg 2048w"
sizes="(max-width: 1024px) 100vw, 1024px" width="1024" height="858"
alt="Knowledge to Action Framework for the Metrics Literacies project at the Scholarly Communications Lab" />
<figcaption>Knowledge to Action (K2A) Framework for the Metric
Literacies project.</figcaption>
</figure>

Scholarly metrics are widely applied to assess research quality and
impact despite their known limitations. One of the most popular
scholarly metrics is the h-index---which is
[defined](https://www.pnas.org/doi/full/10.1073/pnas.0507655102){target="_blank"
rel="noreferrer noopener"} as the "*h* number of papers with at least
*h* number of citations." This means that if a researcher has an h-index
of 12, they have published 12 papers with at least 12 citations each.

Although simple, the h-index is flawed and often applied
inappropriately. It's been criticized due to inconsistencies, lack of
field normalization, database-dependence, and time-dependence. Yet the
h-index continues to be used for hiring, review, and tenure.

[Lauren Maggio](https://www.usuhs.edu/node/10920){target="_blank"
rel="noreferrer noopener"}, professor of medicine at the Uniformed
Services University of Health Sciences, has witnessed how academics
misuse metrics in high-stakes decisions. Across her career, sitting on
many hiring committees, she has "unfortunately seen candidates compared
in relation to their h-index," despite its shortcomings.

"It's really important that people understand how these metrics are
created, that they understand some of the biases and issues that are
baked into these metrics that may disadvantage some groups," she says.
In this post, Lauren tells us about the ScholCommLab's recent work
characterizing available educational videos on the h-index.

<figure class="wp-block-pullquote">
<blockquote>
“It’s really important that people understand how these metrics are
created, that they understand some of the biases and issues that are
baked into these metrics that may disadvantage some groups.”
Lauren Maggio
</blockquote>
</figure>

------------------------------------------------------------------------

## Improving Metrics Literacies through educational videos {#improving-metrics-literacies-through-educational-videos .wp-block-heading}

The project, led by [Stefanie
Haustein](https://www.scholcommlab.ca/stefanie-haustein/){target="_blank"
rel="noreferrer noopener"}, co-director of the ScholCommLab, aims to
improve the appropriate use of scholarly metrics in academia. The team
is developing educational videos and evaluating their efficiency in a
randomized controlled trial to teach academics "[Metrics
Literacies,](https://www.scholcommlab.ca/research/metrics-literacies/){target="_blank"
rel="noreferrer noopener"}" the skills and knowledge to recognize,
interpret, critically assess, and effectively and ethically use
scholarly metrics.

The [multidisciplinary
team](https://www.youtube.com/watch?v=PpAIPEM_HSw){target="_blank"
rel="noreferrer noopener"} consists of bibliometric experts, video
producers, and online education specialists, including Lauren. "Stefanie
asked me to jump on this because of some of the work I've been doing in
the medical education space," she says. They decided to start by
developing open educational resources about the infamous h-index.

Before making their own content, the team assessed the landscape of
existing educational videos on the h-index. By understanding available
resources, Lauren believes that they could "get a sense of how people
were actually talking about the h-index, what it is, issues with it" to
identify content for their own future initiatives. "We could find good
things to do for our own videos or bad things that we knew we wanted to
avoid," she added.

::: {.wp-block-spacer style="height:30px" aria-hidden="true"}
:::

<figure
class="wp-block-embed is-type-rich is-provider-twitter wp-block-embed-twitter">
<div class="wp-block-embed__wrapper">
<blockquote>
Can it get any more meta? Today I presented results from a study of
Youtube videos about the h-index and now I'm on Youtube talking about
the h-index!? <a
href="https://t.co/tor79gY8nq">https://t.co/tor79gY8nq</a> 
w/ <a
href="https://twitter.com/AnitaSamuel28?ref_src=twsrc%5Etfw">@AnitaSamuel28</a>
<a
href="https://twitter.com/LaurenMaggio?ref_src=twsrc%5Etfw">@LaurenMaggio</a>
<a
href="https://twitter.com/alyssajeff12?ref_src=twsrc%5Etfw">@alyssajeff12</a>
<a
href="https://twitter.com/hashtag/MetricsLiteracies?src=hash&amp;ref_src=twsrc%5Etfw">#MetricsLiteracies</a>
<a
href="https://twitter.com/hashtag/ScholCommLab?src=hash&amp;ref_src=twsrc%5Etfw">#ScholCommLab</a>
<a href="https://t.co/4uWI9qyU6W">https://t.co/4uWI9qyU6W</a>
— Stefanie Haustein (@stefhaustein) <a
href="https://twitter.com/stefhaustein/status/1491515073428393997?ref_src=twsrc%5Etfw">February
9, 2022</a>
</blockquote>
</div>
</figure>

::: {.wp-block-spacer style="height:30px" aria-hidden="true"}
:::

In their first study, recently published in [PLOS
ONE](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0268110){target="_blank"
rel="noreferrer noopener"}, the Metrics Literacies team examined the
characteristics of available educational videos on the h-index. They
identified more than 247 videos to get a sense of how the h-index is
described and presented. "We watched a ton of YouTube," laughed Lauren.

Only 31 videos met the inclusion criteria (English-speaking, less than
10 mins long, and actually about the h-index). At least two of the
authors independently coded each of the videos based on their content,
presentation style, and educational quality. 

Specifically, the team used [Cognitive Load
Theory](https://doi.org/10.1207/s15516709cog1202_4){target="_blank"
rel="noreferrer noopener"}---an instructional theory to guide
educational design---to evaluate whether the video format actually
helped people learn. Based on this theory, ideal educational videos
should be designed to balance three types of cognitive load:

- **Reduce extraneous load---**unnecessary design features that prevent
 new information from being processed (e.g., background noises)
- **Minimize intrinsic load---**the inherent difficulty of new
 information. Complex tasks cannot be made easier, but the learning
 burden can be reduced by breaking down content (e.g., segmenting)
- **Increase germane load---**resources required to facilitate learning.
 Presenting content in different ways enhances learning (e.g. using
 audio to explain something, while also showing it with video)

Their results suggest that when it comes to building literacies about
the h-index, the scholarly community has a long way to go. 

------------------------------------------------------------------------

## Most h-index videos use elements to balance cognitive load {#most-h-index-videos-use-elements-to-balance-cognitive-load .wp-block-heading}

<figure class="wp-block-image size-large">
<img
src="https://www.scholcommlab.ca/wp-content/uploads/2022/05/UdgZVvYQ-1024x510.jpeg"
class="wp-image-4039" loading="lazy" decoding="async"
srcset="https://www.scholcommlab.ca/wp-content/uploads/2022/05/UdgZVvYQ-1024x510.jpeg 1024w, https://www.scholcommlab.ca/wp-content/uploads/2022/05/UdgZVvYQ-300x150.jpeg 300w, https://www.scholcommlab.ca/wp-content/uploads/2022/05/UdgZVvYQ-768x383.jpeg 768w, https://www.scholcommlab.ca/wp-content/uploads/2022/05/UdgZVvYQ-1536x765.jpeg 1536w, https://www.scholcommlab.ca/wp-content/uploads/2022/05/UdgZVvYQ-2048x1021.jpeg 2048w"
sizes="(max-width: 1024px) 100vw, 1024px" width="1024" height="510"
alt="Elements used to balance cognitive load (extraneous, intrinsic and germane) from the 31 videos coded and analyzed by the Metrics Literacies team." />
<figcaption aria-hidden="true">Elements used to balance cognitive load
(extraneous, intrinsic and germane) from the 31 videos coded and
analyzed by the Metrics Literacies team.</figcaption>
</figure>

Most of the videos used elements to balance cognitive load. 90% of
videos used signaling (or cues highlighting important details) and 71%
used dual channel approaches (like animated text with voice-over
explanations) to enhance cognitive processing and reduce germane load.
42% of videos actually addressed the viewer, reducing extraneous load. 

Interestingly, only one video included an interactive learning element.
97% of videos did not pause to ask a question, pose a reflection, or
attempt a calculation---which are tools that can facilitate the learning
process. The lack of interaction in the videos could be due to the
nature of the medium, but Lauren believes "It is possible to do those
things. They just weren't taking advantage of that."

The research also made it clear that there is lots of room for when it
comes to minimizing distractions unrelated to learning. 55% of video
included non-productive distractions, which increases extraneous load.
Lauren found the distractions to be extreme in several of the videos.
"You could hear birds, pretty loud, that would just really distract you,
or they would be doing a screencast and just be moving their mouse
everywhere," she recalls. She recommends that future initiatives should
try to be more mindful of reducing these extraneous elements.

## Current videos hardly mention inconsistencies or usefulness of the h-index {#current-videos-hardly-mention-inconsistencies-or-usefulness-of-the-h-index .wp-block-heading}

<figure class="wp-block-image size-large is-resized">
<img
src="https://www.scholcommlab.ca/wp-content/uploads/2022/05/W1_LYxOA-952x1024.jpeg"
class="wp-image-4040" loading="lazy" decoding="async"
srcset="https://www.scholcommlab.ca/wp-content/uploads/2022/05/W1_LYxOA-952x1024.jpeg 952w, https://www.scholcommlab.ca/wp-content/uploads/2022/05/W1_LYxOA-279x300.jpeg 279w, https://www.scholcommlab.ca/wp-content/uploads/2022/05/W1_LYxOA-768x826.jpeg 768w, https://www.scholcommlab.ca/wp-content/uploads/2022/05/W1_LYxOA-1428x1536.jpeg 1428w, https://www.scholcommlab.ca/wp-content/uploads/2022/05/W1_LYxOA-1903x2048.jpeg 1903w"
sizes="(max-width: 714px) 100vw, 714px" width="714" height="768"
alt="Content characteristics of the 31 videos coded and analyzed by the Metrics Literacies team." />
<figcaption aria-hidden="true">Content characteristics of the 31 videos
coded and analyzed by the Metrics Literacies team.</figcaption>
</figure>

While most videos defined or calculated the h-index, 48% of videos did
not mention any caveats. There were no mentions of disadvantages to
early career researchers (77%), database differences (71%), or
disciplinary differences (61%). In fact, Lauren saw that "There were
presenters very much proposing and helping viewers to think about how to
boost their h-index, which shouldn't really have been a surprise." The
lack of critical reflection in the videos fails to provide transparency
on the shortcomings of the h-index.

## Available videos on h-index appear amateurish {#available-videos-on-h-index-appear-amateurish .wp-block-heading}

68% of the videos displayed amateurish production quality. These videos
had poor camera angles, audio quality, and editing, which added
distractions and disrupted the learning experience. In contrast, the
professionally produced videos the team identified flowed smoothly,
incorporating images and animations with scripted dialogue. For the
team, this findings highlights the importance of involving online
education specialists and video producers for developing future videos.

------------------------------------------------------------------------

## A bright future for the Metrics Literacies project {#a-bright-future-for-the-metrics-literacies-project .wp-block-heading}

Thanks to funding from
[SSHRC](https://www.sshrc-crsh.gc.ca/results-resultats/recipients-recipiendaires/2020/ig-ss-eng.aspx){target="_blank"
rel="noreferrer noopener"}, Lauren is excited for what's next: creating
and testing three educational videos on the h-index. "This was really
the background work. Now we're starting to think about how we actually
create the video and how we are going to measure what's happening." 

But the road toward making people metrics literate doesn't stop here.
All of the Metrics Literacies' resources are open through
[Zenodo](https://zenodo.org/communities/metricsliteracies/?page%3D1%26size%3D20&sa=D&source=docs&ust=1651628384962327&usg=AOvVaw03fvL7o3A7EsT-NigvAU3E){target="_blank"
rel="noreferrer noopener"}. Lauren envisions that anyone could adopt
their approaches and materials, using them to improve literacies about
other scholarly metrics. In her words, "If we care about science itself,
and we want science to be sound and ethical, then we really should be
very mindful of what we're doing with our metrics."

***To stay up to date with the*** [***Metrics
Literacies***](https://www.scholcommlab.ca/research/metrics-literacies/){rel="noreferrer noopener"
target="_blank"} ***project, sign up for the*** [***ScholCommLab
newsletter***](https://www.scholcommlab.ca/stay-up-to-date/){rel="noreferrer noopener"
target="_blank"}***. Visit***
[***Zenodo***](https://zenodo.org/communities/metricsliteracies/?page%3D1%26size%3D20&sa=D&source=docs&ust=1651628384962327&usg=AOvVaw03fvL7o3A7EsT-NigvAU3E){rel="noreferrer noopener"
target="_blank"} ***to access all of the Metrics Literacies outputs.***

Scholarly metrics are widely applied to assess research quality and impact despite their known limitations. One of the most popular scholarly metrics is the h-index—which is defined as the “

 h

number of papers with at least

 h

number of citations.” This means that if a researcher has an h-index of 12, they have published 12 papers with at least 12 citations each.

Becoming Metrics Literate: New Study Examines Videos on the H-index

The review, promotion, and tenure (RPT) process is central to academic life and workplace advancement. It influences where faculty direct their attention, research, and publications. By unveiling the RPT process, we can inform actions that lead towards a greater opening of research.

Rethinking Research Assessment for the Greater Good: Findings from the RPT Project

Juan Pablo Alperin

Earlier this year, the ImpactStory team of Heather Piwowar and Jason
Priem launched [Unpaywall](http://unpaywall.org/), a new browser
extension that helps users find free, easy-to-access research.
Since [officially launching in
April](http://blogs.lse.ac.uk/impactofsocialsciences/2017/04/04/announcing-unpaywall-unlocking-openaccess-versions-of-paywalled-research-articles-as-you-browse/),
Unpaywall has been installed by over 85,000 users who have collectively
made over 75 million requests. Like its cousin, the OA Button, and their
illegal counterpart, Sci-Hub, Unpaywall demonstrates the pent-up demand
that exists for access to research.

All three of these tools share something in common: users have found
research they are interested in reading but need a tool to help them
gain access. If academics and non-academics alike are able to find
research they are interested in but need tools such as these for access,
we must ask ourselves: how many more people are being locked out? Or,
put another way, how much are we limiting our capacity to use and
benefit from all the research and scholarship we produce? Fortunately,
data from Unpaywall (or, more precisely, from its underlying
database, [oaDOI](https://oadoi.org/)) can help shed some light on this
question.

In a [recently published preprint](https://peerj.com/preprints/3119/),
the creators of Unpaywall, along with a group of co-authors (myself
included) from three universities, have conducted a large-scale analysis
of the state of open and free access. We found both the number and
proportion of articles that are freely available to the public is
growing, having reached 45% of the literature published in 2015 (Figure
1). While this is not the first study to attempt to measure the
prevalence of OA (see p5 of [our
paper](https://peerj.com/preprints/3119/) for more details of other such
studies), this work stands out because of the size of the sample and the
method of automatically assessing whether free versions of the articles
exist (especially archived copies), and because the underlying data and
service are openly available. In this sense, it is most similar, both in
methods and scale, to the work of [Archambault et al.
(2014)](http://science-metrix.com/sites/default/files/science-metrix/publications/d_1.8_sm_ec_dg-rtd_proportion_oa_1996-2013_v11p.pdf) (whose
work is used to power the 1Science database). Like our own, their study
found approximately half of the scientific papers in their sample were
free to read.

##### ![](https://www.scholcommlab.ca/wp-content/uploads/2017/08/articles-overy-time-1024x340.png){.aligncenter .size-large .wp-image-1016 loading="lazy" decoding="async" width="840" height="279" srcset="https://www.scholcommlab.ca/wp-content/uploads/2017/08/articles-overy-time-1024x340.png 1024w, https://www.scholcommlab.ca/wp-content/uploads/2017/08/articles-overy-time-300x100.png 300w, https://www.scholcommlab.ca/wp-content/uploads/2017/08/articles-overy-time-768x255.png 768w, https://www.scholcommlab.ca/wp-content/uploads/2017/08/articles-overy-time.png 1301w" sizes="(max-width: 840px) 100vw, 840px"}

##### Figure 1. Number of articles (left panel) and proportion of articles (right panel) with OA copies, estimated based on a random sample of 100,000 articles with Crossref DOIs. Gold: published in an open-access journal (as defined by the DOAJ); Green: toll-access on the publisher page, but there is a free copy in an OA repository; Hybrid: free under an open license in a toll-access journal; Bronze: free to read on the publisher page, but without a license; Closed: all other articles, including those shared only on an academic social network or in Sci-Hub. Source: Piwowar H, Priem J, Larivière V, Alperin JP, Matthias L, Norlander B, Farley A, West J, Haustein S. (2017) "The State of OA: A large-scale analysis of the prevalence and impact of Open Access articles", PeerJ Preprints. This work is licensed under a CC BY 4.0 license. {#figure-1.-number-of-articles-left-panel-and-proportion-of-articles-right-panel-with-oa-copies-estimated-based-on-a-random-sample-of-100000-articles-with-crossref-dois.-gold-published-in-an-open-access-journal-as-defined-by-the-doaj-green-toll-access-on-the-publisher-page-but-there-is-a-free-copy-in-an-oa-repository-hybrid-free-under-an-open-license-in-a-toll-access-journal-bronze-free-to-read-on-the-publisher-page-but-without-a-license-closed-all-other-articles-including-those-shared-only-on-an-academic-social-network-or-in-sci-hub.-source-piwowar-h-priem-j-larivière-v-alperin-jp-matthias-l-norlander-b-farley-a-west-j-haustein-s.-2017-the-state-of-oa-a-large-scale-analysis-of-the-prevalence-and-impact-of-open-access-articles-peerj-preprints.-this-work-is-licensed-under-a-cc-by-4.0-license.}

##### ![](http://blogs.lse.ac.uk/impactofsocialsciences/files/2017/08/Figure-1-1.jpg){.aligncenter .size-full .wp-image-31257 loading="lazy" decoding="async" sizes="(max-width: 670px) 100vw, 670px" srcset="http://blogs.lse.ac.uk/impactofsocialsciences/files/2017/08/Figure-1-1.jpg 670w, http://blogs.lse.ac.uk/impactofsocialsciences/files/2017/08/Figure-1-1-300x110.jpg 300w, http://blogs.lse.ac.uk/impactofsocialsciences/files/2017/08/Figure-1-1-669x246.jpg 669w" width="670" height="246" attachment-id="31257" permalink="http://blogs.lse.ac.uk/impactofsocialsciences/2017/08/07/the-number-and-proportion-of-freely-available-articles-is-growing-reaching-45-of-the-literature-published-in-2015/figure-1-21/" orig-file="http://blogs.lse.ac.uk/impactofsocialsciences/files/2017/08/Figure-1-1.jpg" orig-size="670,246" comments-opened="1" image-meta="{\"aperture\":\"0\",\"credit\":\"Booluck,K\",\"camera\":\"\",\"caption\":\"\",\"created_timestamp\":\"1501848718\",\"copyright\":\"\",\"focal_length\":\"0\",\"iso\":\"0\",\"shutter_speed\":\"0\",\"title\":\"\",\"orientation\":\"0\"}" image-title="Figure 1" image-description="" medium-file="http://blogs.lse.ac.uk/impactofsocialsciences/files/2017/08/Figure-1-1-300x110.jpg" large-file="http://blogs.lse.ac.uk/impactofsocialsciences/files/2017/08/Figure-1-1.jpg"}

These studies, and other recent work such as the study of [Sci-Hub usage
data](http://blogs.lse.ac.uk/impactofsocialsciences/2017/06/12/a-closer-look-at-the-sci-hub-corpus-what-is-being-downloaded-and-from-where/),
are just the beginning of a new era of large-scale (dare we say, big
data?!) research on the state of scholarly publishing and the public's
need for research. As we see new open access mandates -- from
universities, funders, and governments alike -- these studies should
make us acutely aware of the value of having open data, along with
high-quality metadata, in machine readable forms. Such data is only now
becoming available. Both the ImpactStory and 1Science teams have had to
resort to developing tools that pool data from many different sources
and work around many of the limitations of the currently available
information.

The result of these efforts are databases -- of which oaDOI has been
made freely and openly available -- that can be used to inform and shape
decisions about what and how researchers and the public are given access
to research.  The value is multiplied when the databases are combined
with usage data from places like Unpaywall, the OA Button, and Sci-Hub,
which allow us to see not only what research is freely available but
what research people are actually trying to access. In the case of
Unpaywall, a sample drawn from a single week of use shows almost 50% of
articles Unpaywall users are interested in reading are freely available
(about a third of these are in OA journals) (Table 1).

##### ![](https://www.scholcommlab.ca/wp-content/uploads/2017/08/Screen-Shot-2018-08-14-at-7.25.36-AM.png){.aligncenter .size-full .wp-image-1019 loading="lazy" decoding="async" width="407" height="265" srcset="https://www.scholcommlab.ca/wp-content/uploads/2017/08/Screen-Shot-2018-08-14-at-7.25.36-AM.png 407w, https://www.scholcommlab.ca/wp-content/uploads/2017/08/Screen-Shot-2018-08-14-at-7.25.36-AM-300x195.png 300w" sizes="(max-width: 407px) 100vw, 407px"}

##### Table 1. Prevalence of OA by type in sample of 100,000 accesses by Unpaywall users {#table-1.-prevalence-of-oa-by-type-in-sample-of-100000-accesses-by-unpaywall-users}

What now seems clear is that publishers are recognising the importance
of providing access. Data from these studies shows an obvious demand for
access, and publishers are responding by launching OA journals, as seen
by the growth of Gold OA content (content published in an open-access
journal, as defined by the [DOAJ](https://doaj.org/)). It can also be
seen in the large proportion of Bronze OA (a new term coined by the
authors, referring to articles that are free to read on the publisher
page, but that have not been explicitly licensed as being open). If we
put on an optimist's lens and view the glass as half-full, it is
certainly encouraging to see our system of scholarly communications
gives users half of the access they need. We can, of course, also view
it from the glass-half-empty perspective, and consider that we still
have more than halfway to go.

What is certain, regardless of whether you are a glass-half-empty or
half-full kind of person, is that we are finally in a position to make
these assessments with some reliability. This study gives us a
reproducible benchmark measure of the extent to which we are enabling
researchers and the public to access scholarly work. Ever the optimist,
I want to believe we will soon be able to quench the world's thirst for
knowledge.

*This blog post is based on the author's co-written article, "*[*The
State of OA: A large-scale analysis of the prevalence and impact of Open
Access articles*](https://peerj.com/preprints/3119/)*", a preprint
currently available at PeerJ Preprints (DOI:
10.7287/peerj.preprints.3119v1). It was originally published at the [LSE
Impact
Blog](http://blogs.lse.ac.uk/impactofsocialsciences/2017/08/07/the-number-and-proportion-of-freely-available-articles-is-growing-reaching-45-of-the-literature-published-in-2015/) under
a [Creative Commons Attribution 3.0 Unported
License](http://creativecommons.org/licenses/by/3.0/deed.en_GB){rel="license"}. *


Earlier this year, the ImpactStory team of Heather Piwowar and Jason Priem launched Unpaywall, a new browser extension that helps users find free, easy-to-access research. Since officially launching in April, Unpaywall has been installed by over 85,000 users who have collectively made over 75 million requests.

Postagens de Rogue Scholar

Becoming Metrics Literate: New Study Examines Videos on the H-index

Rethinking Research Assessment for the Greater Good: Findings from the RPT Project

The number and proportion of freely available articles is growing; reaching 45% of the literature published in 2015